Crawl4AI: Free Open-Source Web Crawler for LLMs

Every AI project hits the same wall: the model reasons brilliantly, but the open web is HTML soup — nav bars, cookie banners, lazy-loaded JavaScript, ads. Dump raw HTML into a context window and you burn thousands of tokens on markup the model ignores. Crawl4AI (73,000+ GitHub stars) exists to end that: URL in, clean Markdown out. Unlike the hosted APIs it competes with, it’s a library you run, not a service you rent.

Is Crawl4AI Really Free?

Yes — completely, for the part almost everyone uses:

• Apache 2.0 license — commercial use, modification, and shipping it inside a paid product, with no revenue cap.
• No key, no account, no quota — you pip install and run. “How many pages a month” is answered by your CPU, not a billing tier.

The one caveat: the maintainers are building a hosted Crawl4AI Cloud API (closed beta, mid-2026) for people who’d rather not run infrastructure. It’s paid, but strictly additive — the self-hosted library stays free and open. That’s the real difference from Firecrawl, whose free tier is a monthly credit allowance that runs out. With Crawl4AI, the free tier is the whole product.

Install and First Crawl (60 Seconds)

Needs Python 3.10+. Two commands — the second downloads the Playwright browser it drives:

pip install -U crawl4ai
crawl4ai-setup

Then the entire “hello world” — fetch a page, print clean Markdown:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://en.wikipedia.org/wiki/Large_language_model")
        print(result.markdown[:1000])

asyncio.run(main())

No key, no config. The result object also carries result.links, result.media, and result.metadata. Prefer the shell? The crwl https://docs.python.org/3/ -o markdown CLI prints Markdown straight to stdout.

The Trick That Makes the Markdown Clean

Anyone can convert HTML to Markdown. What makes Crawl4AI worth feeding to an LLM is its content filters, which throw away boilerplate before the Markdown is generated — that’s where you save tokens. Every crawl produces raw_markdown (the full page) and fit_markdown (the filtered version). You get the filtered one by attaching a filter:

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

md_generator = DefaultMarkdownGenerator(
    content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed")
)
config = CrawlerRunConfig(markdown_generator=md_generator)
# result.markdown.fit_markdown is now the pruned, token-lean version

PruningContentFilter drops low-value chrome (menus, sidebars) — use it for “just the article.” BM25ContentFilter instead ranks blocks against a query you supply (BM25ContentFilter(user_query="pricing and free tier")) — ideal for RAG, where you only want the parts relevant to a question. On content-heavy pages, fit_markdown is a fraction of the raw size — tokens you don’t pay for on every downstream call.

What Else It Does

Beyond a single scrape, the patterns you’ll actually use:

• Batch scrape — arun_many(urls) crawls a list concurrently, with built-in rate limiting and memory-aware dispatch.
• Deep-crawl a whole site — BFSDeepCrawlStrategy(max_depth=2) follows links breadth-first, ideal for ingesting a docs site.
• Structured JSON, no LLM — JsonCssExtractionStrategy pulls records from repeatable layouts via CSS selectors, deterministically and for free.
• LLM extraction, optionally free — LLMExtractionStrategy works with any provider. Point it at a local ollama/llama3.3 and extraction stays $0; swap to a hosted model when you want one.

Run It as a REST API (Docker)

Need to call it over HTTP, from a non-Python service? The official image exposes the whole crawler as a REST API:

docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest
curl -X POST http://localhost:11235/crawl -H "Content-Type: application/json" -d '{"urls": ["https://example.com"]}'

It also serves a /playground UI on port 11235 to test crawls. Drop the container on an always-free VPS and you have a private, unmetered scraping API.

Crawl4AI vs Firecrawl vs Jina Reader

Jina Reader wins on convenience (prefix a URL with r.jina.ai/, zero setup). Firecrawl wins on managed reliability. Crawl4AI wins the moment volume or cost matters — no credit meter, no per-page fee, and an Apache-2.0 license you can bake into a commercial product without AGPL obligations. Many teams prototype on Jina/Firecrawl, then run Crawl4AI in production. It’s also the natural “read the web” layer in a RAG or agent pipeline, pairing with free search, embedding, and vector-DB tiers for a $0 ingestion stack.

Limits to Know

• You own the infrastructure. A headless browser is heavy — expect real RAM/CPU use, especially with arun_many() at high concurrency.
• Anti-bot is an arms race. Stealth and proxy support are built in, but aggressive Cloudflare or login walls can still block you — the exact problem hosted services charge to solve.
• Pre-1.0 API. At v0.9.x it still ships breaking changes between minor versions — pin your version and respect robots.txt.

Frequently Asked Questions

Is Crawl4AI free for commercial use?

Yes. Apache 2.0 permits commercial use, modification, and redistribution with no revenue cap — you can build it into a paid product without open-sourcing your own code, the key advantage over Firecrawl’s AGPL-3.0 self-hosted core.

Does Crawl4AI need an API key?

No. The library needs no key and no account. You only authenticate an LLM if you use the optional LLMExtractionStrategy with a hosted provider — and even that is free with a local Ollama model.

Can it handle JavaScript-heavy sites?

Yes — it drives a real Playwright browser, so it renders single-page apps, waits for dynamic content, and can run custom JS or click/scroll actions before extracting, which a plain HTTP fetch cannot.

How is it different from Scrapy or BeautifulSoup?

Those predate LLMs — you get raw HTML and clean it yourself. Crawl4AI is purpose-built for LLM-ready output: clean Markdown, token-saving filters, and built-in structured/LLM extraction out of the box.

The Verdict

Crawl4AI’s pitch is simple and rare: a production-grade web crawler, engineered for LLMs, that is free in the way that matters — free at scale, free to embed in a product, free of the credit meter every hosted alternative eventually makes you notice. If you’re comfortable running a Python process, it’s the most cost-effective way to give an AI agent clean eyes on the open web. If you want zero ops and will accept a monthly budget, start with Firecrawl; if you just need one page right now, use Jina Reader.

Related Reads

• Firecrawl: Free Web Scraping API for LLMs & Agents
• Tavily vs Brave vs Exa: Free Search APIs for AI Agents
• Cohere Free API: Best Free Embedding and Rerank for RAG
• LiteLLM: One Free API for OpenAI, Claude & 100+ LLMs