crawl4ai

crawl4aiLink to this section

crawl4ai is an agent framework - Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

OverviewLink to this section

Crawl4AI Cloud is a hosted web scraping, extraction, and search API for AI agents and data pipelines. Turn pages into clean Markdown for RAG, extract structured JSON, or search the live web with one API key. Crawl4AI runs the browsers and manages the proxies.

Get a free API key · 1,000 free credits every month. No credit card required.

This repository is home to the Apache-2.0 crawler that powers Crawl4AI Cloud. Run it yourself with Python or Docker.

The same key works for '/extract', '/search', '/serp', '/answer' and many URLs at once ('/scrape/batch', '/scrape/jobs'). Pay as you go: live prices. For Claude Code, connect to the hosted MCP server:

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral.

I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

crawl4ai is open-source, written primarily in Python, with 86,914 GitHub stars under the Apache 2.0 license. The latest release is v0.9.5 (2026-10-14).

Key capabilitiesLink to this section

From the project's documentation:

• LLM-ready output: smart Markdown with headings, tables, code and citation hints
• Fast in practice: async browser pool, caching, minimal hops
• Full control: sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence: learns site patterns, explores only what matters
• 🧹 Clean Markdown: headings, lists, tables and code blocks, in a structure an LLM reads well.
• 🔗 Citations: page links become a numbered reference list.

InstallLink to this section

A quick way to get started (always check the official docs for the latest):

pip install -U crawl4ai

How it fits a local-AI stackLink to this section

crawl4ai runs on your own hardware, so pair it with a model and a GPU sized to your needs. Use the VRAM calculator to pick a model that fits your card, and see what you can run for hardware guidance. Related agent frameworks in the directory:

• "Agent-Reach"
• "agentic-inbox"
• Agno
• "ai-agents-from-scratch"

SourcesLink to this section

• Source code & docs: unclecode/crawl4ai
• Official website: https://crawl4ai.com/?ref=github-about

Stats from GitHub, 2026-11-05.