Did We Give Firecrawl a Fair Test?
We froze a list of 252 URLs before any fetch ran: 12 sites in each of 7 categories, 3 pages a site. Each page went through 12 pipelines, from a plain GET to Firecrawl's API. We counted tokens and checked each output for six-word snippets of the page's real content.
The categories: docs, news and blogs, ecommerce, JS-rendered pages, bot-protected sites, PDFs and small-business pages. We locked the list with a SHA-256 hash and swapped nothing after the first fetch. Firecrawl ran on its defaults, the setup a new user gets.
- Free routes. Raw HTML from a plain GET, BeautifulSoup's
get_texton that HTML, trafilatura with links on and off, and Chromium plus trafilatura. - Hosted readers. Jina Reader without a key, TinyFish Fetch, and Claude Code's WebFetch, which runs a separate model call over the page.
- Firecrawl. The
/v2/scrapemarkdown on defaults, the full JSON response, the markdown with link targets stripped, andonlyMainContentoff on 72 URLs.
We counted tokens with OpenAI's o200k tokenizer. On 10 pages we also measured Claude's count, which ran 1.48 to 1.68 times higher, and the ratios between pipelines moved less than 10%. For quality, we drew up to 10 content probes from each page with a fixed, seeded rule: six-word windows of visible text outside the header, nav and footer, absent from the site's other two pages. A page counts as useful if the fetch worked and the output holds at least 80% of its probes.
Each of the 84 sites carries equal weight. Confidence intervals come from a site bootstrap, and p-values from sign-flip tests with a Holm correction for multiple comparisons.
Does Firecrawl Cut the Tokens You Send a Model?
Against raw HTML, yes. Firecrawl's markdown used 0.060 times the tokens of the page source, a 94% cut. Against free extractors, no. It used 2.91 times the tokens of trafilatura and 1.42 times BeautifulSoup's plain text, and about the same as Jina Reader at 0.96 times.
The intervals hold up: the raw HTML ratio runs from 0.040 to 0.090, and the trafilatura ratio from 2.35 to 3.65. Firecrawl sits mid-field on tokens and at the top on useful pages, 85% on 1,624 tokens.
| Pipeline | Geo-mean tokens | Useful pages | Content recall | Boilerplate leak | Link and wrapper share |
|---|---|---|---|---|---|
| Raw HTML | 26,798 | 77.0% | 83% | 78% | 82% |
| Firecrawl, full JSON | 2,559 | 84.9% | 90% | 35% | 59% |
| Jina Reader | 1,949 | 76.6% | 83% | 51% | 40% |
| Firecrawl markdown | 1,624 | 84.9% | 90% | 35% | 37% |
| Firecrawl, links stripped | 931 | 83.3% | 90% | 35% | 1% |
| BeautifulSoup get_text | 686 | 76.2% | 83% | 77% | 1% |
| TinyFish Fetch | 582 | 63.1% | 73% | 21% | 1% |
| Browser + trafilatura | 540 | 55.2% | 67% | 21% | 3% |
| trafilatura | 487 | 50.4% | 66% | 19% | 1% |
| Claude Code WebFetch | 181 | 7.5% | 8% | 6% | 5% |
Are Those Tokens Content or Link Markup?
About two thirds content. Firecrawl caught 90% of content probes, the top score in the test, against 83% for raw HTML and Jina and 66% for trafilatura. Links and markdown wrapper took 37% of its tokens on the average page, and 59% of the full JSON that API and MCP users receive.
Link targets make up most of that overhead, since markdown writes each link as text plus a URL. We stripped the targets and kept the link text. Firecrawl's tokens fell 43% and its useful rate dropped 1.6 points. If your agent doesn't follow links, you can take that cut for almost nothing.
Firecrawl also carries less site chrome than the free plain-text route: 35% of boilerplate probes against 77% for get_text. The onlyMainContent filter, on by default, does that work. On the 72 URLs we fetched both ways, it cut boilerplate leak from 92% to 39%. The same filter may drop footer details such as a contact email, so test with it off if you scrape for leads.
Does Firecrawl Find the Content More Often?
Yes, by 8.7 points over free plain text, and two categories supply the lead. On ecommerce and JS-rendered pages, Firecrawl beat BeautifulSoup by a wide margin. On docs, news, PDFs and small-business pages, the free route came within 6 points of Firecrawl's useful rate on 74% to 101% of its tokens.
Ecommerce makes the clearest case. Firecrawl made 94% of shop pages useful, against 78% for get_text and 69% for Jina. A plain GET doesn't run JavaScript, so get_text found the content on 3% of JS-rendered pages. Firecrawl reached 50%, and Jina beat it at 61%.
Bot-protected sites went the other way: get_text edged Firecrawl, 81% to 75%. Across the study, Firecrawl refused 16 requests with HTTP 403 and charged nothing for them.
The overall lead needs a caveat. For 21 URLs we had no content probes, so a successful fetch counted as useful. Drop those and Firecrawl's lead over get_text shrinks to 5.7 points, with an interval from about minus 1 to 12.9 that crosses zero. Against Jina, Firecrawl led by 8.3 points, which falls short of significance once we correct for multiple tests (p 0.61).
Do the Credits Pay for Themselves?
Against raw HTML, yes, if your model charges more than $0.054 per million input tokens. A page cost $0.0047 on a Hobby plan with all its credits used and saved about 87,000 tokens. Against trafilatura, no. Firecrawl adds about 3,200 tokens a page, so the credits buy recall and no saving.
Firecrawl's Hobby plan costs $19 a month for 5,000 credits. It charged 1 credit per HTML page and 1 per parsed PDF page, or 1.225 credits per successful page. Use the whole plan and a page costs $0.0047. Use 1,000 credits and it costs $0.0233, five times as much.
Divide the cost of a page by the tokens it saves. At $0.0047 a page and 87,000 tokens saved, Firecrawl pays for itself on any model that charges more than $0.054 per million input tokens, or about $0.037 per million Claude tokens, since Claude counts more tokens for the same text. Half the pages saved under 40,000 tokens, which puts the median page's break-even near $0.12.
Does the Firecrawl MCP Help Inside Claude Code?
It gets more answers right and costs more. On 20 page-fact tasks with Sonnet 5.5, the Firecrawl MCP answered 19 and the built-in WebFetch answered 12. The MCP won 7 tasks that WebFetch lost and lost none (p 0.0156). It used 4.06 times the tokens per task and 3.13 times the Claude dollars.
Each task asked for one fact from one page. The MCP cost $0.0274 of Claude spend a task against $0.0096 for WebFetch, plus 104 Firecrawl credits across the 20 tasks, about 5 a task. Without web tools, Sonnet got none of the 20.
Two things drive the token gap. The Firecrawl MCP server's 28 tool definitions add about 25,000 tokens to each model call, a fixed cost like those we measured in Claude Code's starting context. And WebFetch passes the page to Haiku, so Sonnet reads a digest of 93 tokens at the median, while the MCP hands over the page itself, 1,518 tokens at the median. Median output held near 200 tokens a task in all three setups, so the gap sits on the input side. Count Sonnet's input alone and the MCP used 7.46 times as much.
The page study's 94% cut disappears in Claude Code, because WebFetch shrinks the page further than Firecrawl does. Four of WebFetch's eight misses came from rate limits or a timeout, which a retry might fix. Three had lasting causes: WebFetch blocks the Guardian, and two small-business pages returned a title or the word "Store" and nothing else. Both tools missed the eighth. On the lasting causes alone, the MCP leads 3 to 0, too few to call (p 0.25).
Can You Trust These Numbers?
Our probes check for exact six-word matches, which punishes tools that paraphrase. Claude wrote the probe rule and judged the spot-check, which found 6.8% of probes wrong. Each category carries equal weight, so your own mix of sites will move the averages. The Claude Code result leans on failures a retry might fix.
- Literal probes miss paraphrase. That hurts WebFetch most, since it returns a small model's rewrite.
- Claude graded Claude's rule. In a spot-check of 605 probes, Claude judged 41 wrong (6.8%, CI 5.0% to 9.1%). Counting firm errors alone gives 29, or 4.8%.
- The probe render shares Chromium with the browser pipeline. That pipeline reads the same rendered page the probes came from, which helps its recall.
- Boilerplate probes rest on tags. The rule finds chrome through header, nav and footer tags and through text repeated across a site. Chrome built from plain divs escapes it, so leak figures may run low.
- "Useful" ignores leak and structure. Raw HTML makes 77% of pages useful while 82% of its tokens sit in tags and links.
- Claude token counts are rough. Without an API key, we measured them on a subscription, on Haiku 5.5, over 10 pages.
So, Is Firecrawl Worth It?
Firecrawl earns its price in two cases. You send raw HTML or a browser dump to a model today, or you fetch ecommerce and JS-rendered pages with no browser of your own. For docs, news, PDFs and small-business sites, free plain text comes close on fewer tokens. In Claude Code, the MCP buys accuracy with tokens.
The Claude Code trade comes to 7 more right answers in 20 for 4 times the tokens. If WebFetch fails on the sites you care about, that trade pays. If it doesn't, you carry 25,000 tokens of tool definitions on each call for answers WebFetch would have found.
My Call for Our Own Work
I ran this study to decide if we should pay for Firecrawl, so I checked our own work against those two cases. We send no raw HTML to a model. Our SDR scrapers run their own Chrome, and on small-business pages, their main target, free plain text tied Firecrawl at 92%.
For research I pulled the 247 unique URLs our 38 posts cite: 108 docs pages, 96 news and blog posts, 38 papers, 4 JS or social pages, 1 PDF and no shops. A plain GET with get_text read 228 of the 245 still online, or 93%, counting a page as read at 200 words. Firecrawl recovered 16 of the 17 misses and Jina 8. Eleven misses came from sites such as Forbes, Reuters and Bloomberg that refused the GET with a 401 or 403.
So no, we don't buy a plan. Firecrawl's free tier gives 1,000 credits a month, and the 17 rescue attempts used about 20. In Claude Code I'll keep WebFetch and send its failures to Jina, then Firecrawl, a setup I haven't measured yet. I'll buy a plan if we start feeding raw HTML to a model at volume, scrape shops or JS apps without a browser, or watch the free route's 7% miss rate climb.
My count has two limits. It covers the sources we cite, not a log of all our fetches, and the 200-word bar shows a page loaded without proving it held the fact we wanted. Run the same count on your own fetches before you buy.
TinyFish gets its own post: its Fetch used a third of Firecrawl's tokens and made 22 points fewer pages useful.
