firecrawl

Firecrawl is an open-source project that also offers a hosted service, with the goal of "building the library for superintelligence," augmenting AI agents with data from the web and beyond.

Purpose Firecrawl is used to convert entire websites into Markdown or structured data suitable for use by large language models (LLMs). It is aimed at AI agents and dynamic applications that need real-time web data, helping reduce token consumption and build better AI applications.

Key capabilities

• Industry-leading reliability: covers 96% of web pages, including a large number of JavaScript-dependent pages, without dealing with proxy issues, and directly obtains clean data (official benchmarks provided).
• Speed: P95 latency of 3.4 seconds across millions of pages, aimed at real-time agents and dynamic applications.
• LLM-ready output: clean Markdown, structured JSON, screenshots, and more.
• Handles complex problems: rotating proxies, orchestration, rate limiting, JS-blocked content, etc., with zero configuration.
• Agent-ready: connect Firecrawl to any AI agent or MCP client with a single command.
• Media parsing: parse and extract PDFs, DOCX, and other content hosted on web pages.
• Actions: perform clicks, scrolling, typing, waiting, key presses, and other actions before extracting content.
• Open source: transparent collaborative development, and you can join the community.

Feature overview

• Search the web and retrieve full content from the results.
• Get LLM-ready data from any website: Markdown, JSON, screenshots, and more.
• After scraping a page, interact with it using AI prompts or code.
• Crawl an entire website and retrieve the content of all pages (returns a job ID; the SDK automatically polls).
• Instantly discover all URLs on a website, and also find specific URLs within the site.
• Scrape multiple URLs at once.
• Agent: describe what you need, and the AI agent will search, navigate, and retrieve, without needing to provide URLs. Agent is an evolved version of the /extract endpoint, faster, more reliable, and without needing to know URLs in advance. You can use a schema to obtain structured data, focus the agent on specific pages, and set the level of reasoning effort. Each agent run uses the spark-2 model (default), with no need to set model; the retired spark-1-pro and spark-1-mini names are still accepted for backward compatibility, butDeprecated and running spark-2.

Usage

• Sign up at firecrawl.dev to get an API key, and try it out in the playground.
• Use all features through the SDK, which automatically handles polling for asynchronous operations. After installing the SDK, add the dependency for the corresponding language (Gradle/Maven, etc.).
• Agent integration: After installation, restart the agent; supports Claude Code, Antigravity, OpenCode, etc.; any MCP-compatible client can connect to the network in seconds.
• Agent onboarding: AI agents can get the skill to register for users, obtain an API key, and start using Firecrawl.
• Skills catalog: The Firecrawl Skills Catalog can be installed via npx skills add firecrawl/skills. Build-type skills (integrating Firecrawl into product code) are written under skills/ in this repository and mirrored to the catalog by CI; CLI skills (including the research/developer index skill) are submitted to firecrawl/cli; build/SDK skills are submitted to this repository (skills/); workflow skills are submitted to firecrawl/firecrawl-workflows. The catalog (firecrawl/skills) is read-only; do not submit PRs directly to it.
• For local running, see the Contributing Guide; for self-hosting, see the Self-Hosting Guide.

Use cases

• Provide real-time web data for AI agents.
• Convert website content into LLM-usable Markdown or structured JSON.
• Scenarios that require scraping JS-heavy pages, batch scraping multiple URLs, crawling an entire site, or discovering site URLs.
• Need to extract content from web-hosted media such as PDFs, DOCX, etc.
• Need to interact with the page before extraction (click, scroll, type, wait, press keys).

Limitations and notes

• The open-source version is licensed under AGPL-3.0; the SDK and some UI components are licensed under MIT; see the LICENSE file in each directory for details.
• The cloud version firecrawl.dev includes additional features (the specific contents are not listed in the provided materials).
• Users must comply with website policies, privacy policies, and terms of use on their own; by default, Firecrawl respects robots.txt directives.
• Regarding specific charges, pricing, platform support scope, etc., the materials do not currently provide this.