Firecrawl V1 is here! With that we introduce a more reliable and developer friendly API.

August 29th, 2024

Here is what’s new:

• Output Formats for /scrape. Choose what formats you want your output in.
• New /map endpoint for getting most of the URLs of a webpage.
• Developer friendly API for /crawl/{id} status.
• 2x Rate Limits for all plans.
• Go SDK and Rust SDK
• Teams support
• API Key Management in the dashboard.
• onlyMainContent is now default to true.
• /crawl webhooks and websocket support.

Learn more about it here

Start using v1 right away at https://firecrawl.dev

What's Changed (including v0 + v1)

• Delete .DS_Store by @szepeviktor in Delete .DS_Store #8
• [Bugfix] added normalized apikey to craw/status route by @rafaelsideguide in [Bugfix] added normalized apikey to craw/status route #12
• [Feat] improving reative paths by @rafaelsideguide in [Feat] improving reative paths #4
• Fix typos by @szepeviktor in Fix typos #9
• [Feat] Added html to markdown table parser by @rafaelsideguide in [Feat] Added html to markdown table parser #11
• Option to extract only the main content, excluding headers, navs, footers etc. by @nickscamara in Option to extract only the main content, excluding headers, navs, footers etc. #14
• [Feat] Adding pdf parser by @rafaelsideguide in [Feat] Adding pdf parser #17
• adding ci-cd workflow by @rafaelsideguide in adding ci-cd workflow #20
• adding workflow by @rafaelsideguide in adding workflow #21
• adding env secrets by @rafaelsideguide in adding env secrets #22
• [Feat] Added TSDocs and types for js-sdk by @rafaelsideguide in [Feat] Added TSDocs and types for js-sdk #28
• Added option to replace all relative paths with absolute paths by @rafaelsideguide in Added option to replace all relative paths with absolute paths #25
• [Bugfix] Fixed scrape preview test by @rafaelsideguide in [Bugfix] Fixed scrape preview test #30
• Caleb: fixing some documentation and rebuilding the server by @calebpeffer in Caleb: fixing some documentation and rebuilding the server #32
• Rate limit fixes for crawl status by @nickscamara in Rate limit fixes for crawl status #36
• Better logging by @nickscamara in Better logging #35
• [Feat] Added type declarations by @rafaelsideguide in [Feat] Added type declarations #31
• Refactor api routes by @nickscamara in Refactor api routes #37
• Logging by @nickscamara in Logging #38
• Cjp/making db auth optional <> Running project locally by @calebpeffer in Cjp/making db auth optional <> Running project locally #40
• chore: add context.close by @mattzcarey in chore: add context.close #46
• Fixes table parsing for websites such as news.ycombinator.com (HN) by @nickscamara in Fixes table parsing for websites such as news.ycombinator.com (HN) #52
• [Feat] Server health check + slack message by @rafaelsideguide in [Feat] Server health check + slack message #53
• [Feat] Added blocklist for social media urls by @rafaelsideguide in [Feat] Added blocklist for social media urls #55
• [Feat:mvp] Search Endpoint => serp api + firecrawl => 🔥 🔍 by @nickscamara in [Feat:mvp] Search Endpoint => serp api + firecrawl => 🔥 🔍  #56
• [Feat] Added anthropic vision api by @rafaelsideguide in [Feat] Added anthropic vision api #5
• [Bugfix] Trim and Lowercase all urls by @rafaelsideguide in [Bugfix] Trim and Lowercase all urls #13
• Implements the ability for the crawler to output all the links it found, without scraping by @nickscamara in Implements the ability for the crawler to output all the links it found, without scraping #34
• Serper params by @nickscamara in Serper params #62
• Support for tbs, filter, lang, country and location with Serper search. by @rogerserper in Support for tbs, filter, lang, country and location with Serper search. #61
• [Feat] Added allowed urls by @rafaelsideguide in [Feat] Added allowed urls #64
• /search support in node sdk by @nickscamara in /search support in node sdk #72
• Free credits increase by @nickscamara in Free credits increase #75
• [Bugfix] JS-SDK: Remove dotenv and add tests by @mdp in [Bugfix] JS-SDK: Remove dotenv and add tests #68
• [Feat] Coupon system by @rafaelsideguide in [Feat] Coupon system #66
• Specific website params support by @nickscamara in Specific website params support #83
• Greenpay fixes by @nickscamara in Greenpay fixes #84
• [Feat] Implemented retry attempts to handle 502 errors by @rafaelsideguide in [Feat] Implemented retry attempts to handle 502 errors #67
• feat: LLM Extraction (mvp) by @nickscamara in feat: LLM Extraction (mvp) #90
• Update README.md by @bllchmbrs in Update README.md #110
• Add Posthog Logging by @ericciarla in Add Posthog Logging #109
• Refactor of main web scraper + Partial data streaming by @nickscamara in Refactor of main web scraper + Partial data streaming #120
• [Feat] Added includeHTML option by @rafaelsideguide in [Feat] Added includeHTML option #126
• Cancel Job Route by @nickscamara in Cancel Job Route #129
• [Feat] Added max depth option by @rafaelsideguide in [Feat] Added max depth option #130
• Add keyAuth endpoint by @ericciarla in Add keyAuth endpoint #131
• [Test] Added integration tests suite by @rafaelsideguide in [Test] Added integration tests suite #118
• Adds Zod Integration for LLM Extraction in the Firecrawl JS SDK by @nickscamara in Adds Zod Integration for LLM Extraction in the Firecrawl JS SDK  #135
• [Docs] Updated examples by @rafaelsideguide in [Docs] Updated examples #137
• Switching to AGPL - We Need Your Consent! by @calebpeffer in Switching to AGPL - We Need Your Consent! #134
• Nsc/refactor scraping order by @nickscamara in Nsc/refactor scraping order #139
• Update models.ts by @ericciarla in Update models.ts #144
• Timeout on /scrape by @nickscamara in Timeout on /scrape #145
• [Doc] Added default value for crawlOptions.limit by @rafaelsideguide in [Doc] Added default value for crawlOptions.limit #142
• feat: 4x-5x faster crawler (fast mode) by @nickscamara in feat: 4x-5x faster crawler (fast mode) #149
• Add Docker Compose for easy self hosting by @chand1012 in Add Docker Compose for easy self hosting #119
• refactor: fix typo in WebScraper/index.ts by @eltociear in refactor: fix typo in WebScraper/index.ts #27
• [Tests] Added crawl test suite -> crawl improvements by @rafaelsideguide in [Tests] Added crawl test suite -> crawl improvements #153
• feat: Docx Support by @nickscamara in feat: Docx Support #158
• Fixes pdfs not found if .pdf is not present by @nickscamara in Fixes pdfs not found if .pdf is not present #29
• Update README.md: Typo fix by @elimisteve in Update README.md: Typo fix #160
• [Feat] Added rate limits by @rafaelsideguide in [Feat] Added rate limits #151
• Allow override of API URL by @mattjoyce in Allow override of API URL #166
• feat: HyperDX Integration by @nickscamara in feat: HyperDX Integration #167
• beta: Fire-Engine fallback by @nickscamara in beta: Fire-Engine fallback #174
• Add additional file extensions to crawler.ts by @tractorjuice in Add additional file extensions to crawler.ts #77
• [Bug] Fixing /crawl limit by @rafaelsideguide in [Bug] Fixing /crawl limit #143
• Update issue templates by @rafaelsideguide in Update issue templates #180
• [Feat] Added proxy and media blocking support for Playwright by @JakobStadlhuber in [Feat] Added proxy and media blocking support for Playwright #181
• update: wait until body attached in playwright-service by @qyou in update: wait until body attached in playwright-service #170
• feat: Allow privacy/legal/ other pages in social media websites by @nickscamara in feat: Allow privacy/legal/ other pages in social media websites  #168
• [Bug] Added data check for python SDK by @rafaelsideguide in [Bug] Added data check for python SDK #176
• Fix FIRECRAWL_API_URL bug, also various PyLint fixes by @mattjoyce in Fix FIRECRAWL_API_URL bug, also various PyLint fixes #178
• [Feat] Added idempotency key to crawl route by @rafaelsideguide in [Feat] Added idempotency key to crawl route #132
• Feat: Provide more details for 429 error msg by @simonha9 in Feat: Provide more details for 429 error msg #190
• Limit on /search is not deterministic by @Keredu in Limit on /search is not deterministic #186
• Various PyPi Metadata by @mattjoyce in Various PyPi Metadata #191
• [Test] Added sdk e2e tests by @rafaelsideguide in [Test] Added sdk e2e tests #183
• Allow users to manually set the waitFor param on /scrape by @nickscamara in Allow users to manually set the waitFor param on /scrape #200
• [Feat] Added custom scraping conditions for readme docs by @rafaelsideguide in [Feat] Added custom scraping conditions for readme docs #204
• Feat/screenshot support by @ericciarla in Feat/screenshot support #207
• feat: New pricing/limits changes by @nickscamara in feat: New pricing/limits changes #216
• [sdk] Fixes waiting status not being present on check status by @nickscamara in [sdk] Fixes waiting status not being present on check status #218
• Fixed fire-engine content bug by @rafaelsideguide in Fixed fire-engine content bug #228
• Use @ instead of # for default BULL_AUTH_KEY. Hash mark is reserved for URI fragments. by @rombru in Use @ instead of # for default BULL_AUTH_KEY. Hash mark is reserved for URI fragments. #229
• feat: Ability to forward headers to reliable providers for auth etc... by @nickscamara in feat: Ability to forward headers to reliable providers for auth etc... #221
• Improvements to the blocklist regex by @nickscamara in Improvements to the blocklist regex #185
• Playwright service bugs [BUG] Playwright service is crashing - AttributeError: 'UrlModel' object has no attribute 'timeout' #222 Falling back to fetch problem #179 [BUG] Search is returning "no page found" on self-host #197 by @mattjoyce in Playwright service bugs #222 #179 #197 #224
• [Fix] Changed timeout parameter name on js sdk by @rafaelsideguide in [Fix] Changed timeout parameter name on js sdk #211
• [Bug] Improved js response and test for getting partial_data by @rafaelsideguide in [Bug] Improved js response and test for getting partial_data #212
• Better fallbacks for initial crawl start by @nickscamara in Better fallbacks for initial crawl start #148
• [Feat] Added custom scraping for google-drive pdf usecase by @rafaelsideguide in [Feat] Added custom scraping for google-drive pdf usecase #235
• Script to check local vs published versions of sdk packages by @mattjoyce in Script to check local vs published versions of sdk packages #213
• Update README.md by @calebpeffer in Update README.md #140
• Refactor custom scraping and added security center vanta by @nickscamara in Refactor custom scraping and added security center vanta #237
• [Feat] Added scroll xpaths on fire-engine for handling readme docs by @rafaelsideguide in [Feat] Added scroll xpaths on fire-engine for handling readme docs #239
• Add Kubernetes configuration for Firecrawl deployment by @JakobStadlhuber in Add Kubernetes configuration for Firecrawl deployment #236
• [Feat] Improved the scrape for gdrive pdfs by @rafaelsideguide in [Feat] Improved the scrape for gdrive pdfs #238
• Add support for Self-Hosted Webhook URL Usage and added project_id into the webhook payload by @JakobStadlhuber in Add support for Self-Hosted Webhook URL Usage and added project_id into the webhook payload #234
• Partial Data Sliding window of 50 by @nickscamara in Partial Data Sliding window of 50 #242
• feat: Transactional emails for rate limits and credit limits by @nickscamara in feat: Transactional emails for rate limits and credit limits #244
• Python-SDK transitional build setup for pyproject.toml by @mattjoyce in Python-SDK transitional build setup for pyproject.toml #196
• [Feat] CI/CD for publishing js and python SDKs by @rafaelsideguide in [Feat] CI/CD for publishing js and python SDKs #246
• 194 sdk ci pipeline for publishing pythonnode sdk by @rafaelsideguide in 194 sdk ci pipeline for publishing pythonnode sdk #248
• fixing minor problems on workflow by @rafaelsideguide in fixing minor problems on workflow #250
• Fix 208 py sdk interval poll name by @mattjoyce in Fix 208 py sdk interval poll name #252
• Moving from fetch to axios and preventing deadlocks by setting timeouts on fallbacks by @nickscamara in Moving from fetch to axios and preventing deadlocks by setting timeouts on fallbacks #260
• ignoreSitemap feature, pageOptions now respected in the initial crawl as well by @nickscamara in ignoreSitemap feature, pageOptions now respected in the initial crawl as well #263
• Fixed bugs associated with absolute path replacements by @nickscamara in Fixed bugs associated with absolute path replacements #268
• Only fetch webhook from db if self host webhook not set and using db auth by @nickscamara in Only fetch webhook from db if self host webhook not set and using db auth #262
• [Feat] Added route to clean completed jobs and a github action cron that triggers every 24h by @rafaelsideguide in [Feat] Added route to clean completed jobs and a github action cron that triggers every 24h #265
• [Feat] Added allowBackwardCrawling option by @rafaelsideguide in [Feat] Added allowBackwardCrawling option #269
• [Feat] Added jobId to webhook data by @rafaelsideguide in [Feat] Added jobId to webhook data #272
• Clusters support to scale our API to # of CPUs running by @nickscamara in Clusters support to scale our API to # of CPUs running #274
• Added pageOptions.removeTags by @rafaelsideguide in Added pageOptions.removeTags #275
• Added logging to python sdk FIRECRAWL_LOGGING_LEVEL by @mattjoyce in Added logging to python sdk FIRECRAWL_LOGGING_LEVEL #256
• Python sdk improve response handling by @mattjoyce in Python sdk improve response handling #253
• [Feat] Added parsePDF option to pageOptions by @rafaelsideguide in [Feat] Added parsePDF option to pageOptions #271
• [Feat] Added metadata.pageStatusCode and metadata.pageError properties by @rafaelsideguide in [Feat] Added metadata.pageStatusCode and metadata.pageError properties #276
• Make maxDepth relative to the entered url by @ericciarla in Make maxDepth relative to the entered url #277
• Test/load testing by @rafaelsideguide in Test/load testing #175
• Fixes crawler getting confused with base paths that contain www. by @nickscamara in Fixes crawler getting confused with base paths that contain www. #283
• Feat/max depth relative by @ericciarla in Feat/max depth relative #285
• [Feat] Improvements on response document types by @rafaelsideguide in [Feat] Improvements on response document types #298
• test: Rate Limit Unit Tests by @nickscamara in test: Rate Limit Unit Tests #282
• Added local host support for the javascript SDK by @NeevJewalkar in Added local host support for the javascript SDK #296
• [Test] Transcribed from e2e to unit tests for many cases by @rafaelsideguide in [Test] Transcribed from e2e to unit tests for many cases #294
• [Feat] Added support for RegEx in removeTags by @AndyMik90 in [Feat] Added support for RegEx in removeTags #297
• [Bug] Fixed includeHTML to use cleanedHtml as response by @rafaelsideguide in [Bug] Fixed includeHTML to use cleanedHtml as response #301
• Cjp/email to posthog logging by @calebpeffer in Cjp/email to posthog logging #302
• Fix Broken Link by @Lakr233 in Fix Broken Link #303
• docs: Fix pydanti to pydantic by @100gle in docs: Fix pydanti to pydantic #308
• [Bug] Fixed axios bug that were making jobs stuck on active queue by @rafaelsideguide in [Bug] Fixed axios bug that were making jobs stuck on active queue #312
• fix multi-word search term issue: /search (w/o Serp) by @snippet in fix multi-word search term issue: /search (w/o Serp) #313
• [Bug] Fixed clean jobs by @rafaelsideguide in [Bug] Fixed clean jobs #317
• [Bug] Fixed the regex test for google drive pdf files by @rafaelsideguide in [Bug] Fixed the regex test for google drive pdf files #318
• add some types by @snippet in add some types #316
• [Bug] Added default values and fixed pdf bug by @rafaelsideguide in [Bug] Added default values and fixed pdf bug #321
• [Test] Added E2E tests for checking metadata values by @rafaelsideguide in [Test] Added E2E tests for checking metadata values #322
• pageOptions.onlyIncludeTags param by @nickscamara in pageOptions.onlyIncludeTags param #328
• Update CONTRIBUTING.md by @george-zakharov in Update CONTRIBUTING.md #329
• Fix headers | pageOptions not being passed correctly to fire engine by @nickscamara in Fix headers | pageOptions not being passed correctly to fire engine #331
• [Tests] Added crawl limit unit test by @rafaelsideguide in [Tests] Added crawl limit unit test #323
• Adds pageOptions.includeRawHtml and new extraction mode "llm-extraction-from-raw-html" by @ericciarla in Adds pageOptions.includeRawHtml and new extraction mode "llm-extraction-from-raw-html" #332
• [Proposal] new feature allowExternalContentLinks by @snippet in [Proposal] new feature allowExternalContentLinks #336
• [PROPOSAL] (deps): making sure all deps are always up to date by @Sanix-Darker in [PROPOSAL] (deps): making sure all deps are always up to date #338
• [PROPOSAL] (docker-compose) regroup envs vars between services by @Sanix-Darker in [PROPOSAL] (docker-compose) regroup envs vars between services  #337
• new playwright service by @snippet in new playwright service #325
• apps/playwright-service(deps): bump the prod-deps group in /apps/playwright-service with 3 updates by @dependabot in apps/playwright-service(deps): bump the prod-deps group in /apps/playwright-service with 3 updates #346
• Logging for all scraper methods by @nickscamara in Logging for all scraper methods #363
• setting up docker to ts playwright service by @snippet in setting up docker to ts playwright service #361
• (Docs) Self Host added new ts playwright service instructions by @snippet in (Docs) Self Host added new ts playwright service instructions #362
• apps/test-suite(deps): bump the prod-deps group in /apps/test-suite with 6 updates by @dependabot in apps/test-suite(deps): bump the prod-deps group in /apps/test-suite with 6 updates #347
• apps/test-suite(deps-dev): bump typescript from 5.4.5 to 5.5.3 in /apps/test-suite in the dev-deps group by @dependabot in apps/test-suite(deps-dev): bump typescript from 5.4.5 to 5.5.3 in /apps/test-suite in the dev-deps group #348
• apps/api(deps): bump the prod-deps group in /apps/api with 28 updates by @dependabot in apps/api(deps): bump the prod-deps group in /apps/api with 28 updates #349
• [Feat] Added veeva to special case params by @rafaelsideguide in [Feat] Added veeva to special case params #371
• Only check Supabase if configured to when cancelling job by @StefanTerdell in Only check Supabase if configured to when cancelling job #374
• dependabot for security checks, fixed crawl test by @rafaelsideguide in dependabot for security checks, fixed crawl test #370
• [Feat] Added implementation for saving docs on supabase by @rafaelsideguide in [Feat] Added implementation for saving docs on supabase #326
• Self-hosting quality of life fixes by @StefanTerdell in Self-hosting quality of life fixes #375
• [Feat] Added fire-engine fallback for getting sitemaps by @rafaelsideguide in [Feat] Added fire-engine fallback for getting sitemaps #386
• fix(bull): requeue jobs after restart by @mogery in fix(bull): requeue jobs after restart #393
• Fix USE_DB_AUTHENTICATION checks for self-host by @kun432 in Fix USE_DB_AUTHENTICATION checks for self-host #395
• [BUG] Fixed docker self hosting issue by @rafaelsideguide in [BUG] Fixed docker self hosting issue #396
• Slack Alerts for when queue is over capacity by @nickscamara in Slack Alerts for when queue is over capacity #397
• CI for alerts instead by @nickscamara in CI for alerts instead #398
• Log Fire-engine page errors by @nickscamara in Log Fire-engine page errors #394
• Sitemap fallback fixes w/ fire-engine by @nickscamara in Sitemap fallback fixes w/ fire-engine #399
• Separate Rate Limiter from Main Redis by @nickscamara in Separate Rate Limiter from Main Redis #417
• [Feat] Pass along current, total, current_step, and current_url in js sdk by @jhoseph88 in [Feat] Pass along current, total, current_step, and current_url in js sdk #391
• Redis Health Checks by @nickscamara in Redis Health Checks #424
• Small fix for empty pageOptions by @rafaelsideguide in Small fix for empty pageOptions #420
• Caleb: Return a list of links on a page by default by @calebpeffer in Caleb: Return a list of links on a page by default #423
• [Docs] Updating docs by @rafaelsideguide in [Docs] Updating docs #427
• Fix queue stuck bug via lock settings changes by @mogery in Fix queue stuck bug via lock settings changes #429
• fix(js-sdk): transform tests with ts-jest and configure node by @mogery in fix(js-sdk): transform tests with ts-jest and configure node #433
• fix(WebScraper): infinite regex leading to fly.io instance hangs by @mogery in fix(WebScraper): infinite regex leading to fly.io instance hangs #436
• Support chrome-cdp and restructure sitemap fire-engine support. by @tomkosm in Support chrome-cdp and restructure sitemap fire-engine support. #410
• /scrape should now be 600ms-900ms faster by @nickscamara in /scrape should now be 600ms-900ms faster #447
• Notion Website Fixes by @nickscamara in Notion Website Fixes #453
• Firecrawl UI template by @ericciarla in Firecrawl UI template #458
• Added regex for links in sitemap by @rafaelsideguide in Added regex for links in sitemap #449
• Added logger by @rafaelsideguide in Added logger #450
• Add scrape monitoring by @mogery in Add scrape monitoring #455
• Readiness liveness probes by @JakobStadlhuber in Readiness liveness probes #457
• Client side error handling by @nickscamara in Client side error handling #461
• Admin router + Improve redis notifications by @nickscamara in Admin router + Improve redis notifications #460
• support custom models by @NiuBlibing in support custom models #414
• DNS Cacheable Lookup to avoid DNS blocking ops by @nickscamara in DNS Cacheable Lookup to avoid DNS blocking ops #478
• Fire-engine to be the default for our cloud-service by @nickscamara in Fire-engine to be the default for our cloud-service #479
• [Bug] Rate limiter bug on switch for some cases by @rafaelsideguide in [Bug] Rate limiter bug on switch for some cases #482
• [Feat] Better self hosting guide by @rafaelsideguide in [Feat] Better self hosting guide #483
• Fix parallelization issues with caching by @nickscamara in Fix parallelization issues with caching #484
• [Bug] Issue with crawl going beyond Limit by @rafaelsideguide in [Bug] Issue with crawl going beyond Limit #485
• [Bug] pdfs and logging pdf events, also added trycatchs for docx by @rafaelsideguide in [Bug] pdfs and logging pdf events, also added trycatchs for docx #475
• [Bug] Nested sitemaps by @rafaelsideguide in [Bug] Nested sitemaps #491
• Fix AMZN | Removal of redis alerts by @nickscamara in Fix AMZN | Removal of redis alerts #492
• Custom engine params fix by @nickscamara in Custom engine params fix #494
• [Bug] Fixed the empty excludes.filter is undefined bug by @rafaelsideguide in [Bug] Fixed the empty excludes.filter is undefined bug #503
• [Feat] Added fullpagescreenshot capabilities by @rafaelsideguide in [Feat] Added fullpagescreenshot capabilities #504
• fix(js-sdk): build both CommonJS and ESM versions by @mogery in fix(js-sdk): build both CommonJS and ESM versions #432
• fix(js-sdk): build both CJS and EJS versions by @mogery in fix(js-sdk): build both CJS and EJS versions #508
• Improve logs by @tak-s in Improve logs #496
• Digikey support by @nickscamara in Digikey support #512
• [Feat] Add Go SDK implementation by @KentHsu in [Feat] Add Go SDK implementation #497
• Redlock cache in auth by @nickscamara in Redlock cache in auth #529
• Update redis urls in example .env by @wahpiangle in Update redis urls in example .env #515
• Self-host fix: Moving comments of .env.example values from end-of-line to above-line. by @kevinswiber in Self-host fix: Moving comments of .env.example values from end-of-line to above-line. #531
• fix: go-sdk module name by @KentHsu in fix: go-sdk module name #534
• Fixed e2e tests by @rafaelsideguide in Fixed e2e tests #536
• Removed obsoleted declaration by @matsubo in Removed obsoleted declaration #532
• [Feat] Added go-sdk as submodule by @rafaelsideguide in [Feat] Added go-sdk as submodule #537
• feat: Move scraper to queue by @mogery in feat: Move scraper to queue #459
• Reduce metrics ingestion w/ HyperDX v0.8.1 by @nickscamara in Reduce metrics ingestion w/ HyperDX v0.8.1 #480
• Check team credits based on the crawl limit by @nickscamara in Check team credits based on the crawl limit #554
• added variables to beta customers by @rafaelsideguide in added variables to beta customers #555
• Set the crawl limit to the remaining credits by @nickscamara in Set the crawl limit to the remaining credits #559
• [Feat] Added check job and cancel to fire-engine requests by @rafaelsideguide in [Feat] Added check job and cancel to fire-engine requests #560
• [Bug] Added a way for dealing with DNS without IP resolution by @rafaelsideguide in [Bug] Added a way for dealing with DNS without IP resolution #561
• Added Sentry Monitoring by @nickscamara in Added Sentry Monitoring #562
• Internal Concurrency Limits <> Job Priorities by @nickscamara in Internal Concurrency Limits <> Job Priorities #566
• [V1] Release by @rafaelsideguide in [V1] Release #527
• [Feat]: Add RUST SDK client for firecrawl API by @Sanix-Darker in [Feat]: Add RUST SDK client for firecrawl API  #373
• Update README.md by @ericciarla in Update README.md #590
• [Bug] Moved delete rawHtml to end of controller by @rafaelsideguide in [Bug] Moved delete rawHtml to end of controller #582
• Fixed RPC cloudflare issues with job ids by @nickscamara in Fixed RPC cloudflare issues with job ids #593
• [v1] Websockets SDKs by @rafaelsideguide in [v1] Websockets SDKs #591
• [v1] LLM Extract by @nickscamara in [v1] LLM Extract #586
• [v1] Webhooks by @rafaelsideguide in [v1] Webhooks #594
• fix(v1): maxDepth by @rafaelsideguide in fix(v1): maxDepth #607
• Ensuring USE_DB_AUTHENTICATION is true in single URL scraper. by @kevinswiber in Ensuring USE_DB_AUTHENTICATION is true in single URL scraper. #516
• Bill team async by @nickscamara in Bill team async #619
• Optimize check credits func w/ caching by @nickscamara in Optimize check credits func w/ caching #624
• Feat: Added go html to md parser by @rafaelsideguide in Feat: Added go html to md parser #608
• Feat: parser singleton by @rafaelsideguide in Feat: parser singleton #629

New Contributors

• @szepeviktor made their first contribution in Delete .DS_Store #8
• @rafaelsideguide made their first contribution in [Bugfix] added normalized apikey to craw/status route #12
• @nickscamara made their first contribution in Option to extract only the main content, excluding headers, navs, footers etc. #14
• @calebpeffer made their first contribution in Caleb: fixing some documentation and rebuilding the server #32
• @mattzcarey made their first contribution in chore: add context.close #46
• @rogerserper made their first contribution in Support for tbs, filter, lang, country and location with Serper search. #61
• @mdp made their first contribution in [Bugfix] JS-SDK: Remove dotenv and add tests #68
• @bllchmbrs made their first contribution in Update README.md #110
• @ericciarla made their first contribution in Add Posthog Logging #109
• @chand1012 made their first contribution in Add Docker Compose for easy self hosting #119
• @eltociear made their first contribution in refactor: fix typo in WebScraper/index.ts #27
• @elimisteve made their first contribution in Update README.md: Typo fix #160
• @mattjoyce made their first contribution in Allow override of API URL #166
• @tractorjuice made their first contribution in Add additional file extensions to crawler.ts #77
• @JakobStadlhuber made their first contribution in [Feat] Added proxy and media blocking support for Playwright #181
• @qyou made their first contribution in update: wait until body attached in playwright-service #170
• @simonha9 made their first contribution in Feat: Provide more details for 429 error msg #190
• @Keredu made their first contribution in Limit on /search is not deterministic #186
• @rombru made their first contribution in Use @ instead of # for default BULL_AUTH_KEY. Hash mark is reserved for URI fragments. #229
• @NeevJewalkar made their first contribution in Added local host support for the javascript SDK #296
• @AndyMik90 made their first contribution in [Feat] Added support for RegEx in removeTags #297
• @Lakr233 made their first contribution in Fix Broken Link #303
• @100gle made their first contribution in docs: Fix pydanti to pydantic #308
• @snippet made their first contribution in fix multi-word search term issue: /search (w/o Serp) #313
• @george-zakharov made their first contribution in Update CONTRIBUTING.md #329
• @Sanix-Darker made their first contribution in [PROPOSAL] (deps): making sure all deps are always up to date #338
• @dependabot made their first contribution in apps/playwright-service(deps): bump the prod-deps group in /apps/playwright-service with 3 updates #346
• @StefanTerdell made their first contribution in Only check Supabase if configured to when cancelling job #374
• @mogery made their first contribution in fix(bull): requeue jobs after restart #393
• @kun432 made their first contribution in Fix USE_DB_AUTHENTICATION checks for self-host #395
• @jhoseph88 made their first contribution in [Feat] Pass along current, total, current_step, and current_url in js sdk #391
• @tomkosm made their first contribution in Support chrome-cdp and restructure sitemap fire-engine support. #410
• @NiuBlibing made their first contribution in support custom models #414
• @tak-s made their first contribution in Improve logs #496
• @KentHsu made their first contribution in [Feat] Add Go SDK implementation #497
• @wahpiangle made their first contribution in Update redis urls in example .env #515
• @kevinswiber made their first contribution in Self-host fix: Moving comments of .env.example values from end-of-line to above-line. #531
• @matsubo made their first contribution in Removed obsoleted declaration #532

Full Changelog: https://github.com/mendableai/firecrawl/commits/v1.0.0