firecrawl

Firecrawl is an open source project aimed at providing data from the web and broader sources to AI agents, helping to build a superintelligence library. It is both open source and available as a hosted service.

Purpose: Firecrawl's primary purpose is to convert entire websites into LLM-ready Markdown or structured data, enabling AI agents and dynamic applications to obtain real-time web data. It achieves scraping, crawling, and extraction through a single API.

Key capabilities:

• Industry-leading reliability: covers 96% of web pages, including JS-intensive pages, with no need to handle proxy issues and direct access to clean data (see benchmarks).
• Extremely fast: P95 latency of 3.4 seconds across millions of pages, built for real-time agents and dynamic applications.
• LLM-ready output: provides clean Markdown, structured JSON, screenshots, etc., reducing token consumption and building better AI applications.
• Handles complexity: automatically handles rotating proxies, orchestration, rate limits, JS-blocked content, etc., with zero configuration.
• Agent-ready: connect Firecrawl to any AI agent or MCP client with a single command.
• Media parsing: parse and extract content from web-hosted files such as PDF and DOCX.
• Actions: perform actions such as click, scroll, write, wait, and key press before extracting content.
• Open source: transparent and collaborative development, you can join the community.

How to use:

• Sign up at firecrawl.dev to get an API key, and you can test it in the playground.
• Search the web and get full content from the results.
• Get LLM-ready data from any website—Markdown, JSON, screenshots, etc.
• After scraping a page, interact with it using AI prompts or code.
• Connect AI agents: connect Firecrawl to any AI agent or MCP client with a single command, and restart the agent after installation. Works with Claude Code, Antigravity, OpenCode, etc. Any MCP-compatible client can connect to the web in seconds.
• Agent onboarding: AI agents can obtain skills to register users, get API keys, and start using Firecrawl. See the skills + CLI documentation for all commands; for MCP, see firecrawl-mcp-server.
• Agent features: Describe what is needed, and the AI agent will search, navigate, and retrieve, with no need to provide a URL. Agent is the evolution of the /extract endpoint, faster, more reliable, and requiring no prior knowledge of the URL. A schema can be used to obtain structured data, focus on specific pages, and set the amount of reasoning the agent spends on the task. Every effort level runs the spark-2 model; what effort changes is the reasoning budget, not the model. By default, each agent run executes on spark-2, with no need to set a model. The deprecated spark-1-pro and spark-1-mini names are still accepted for backward compatibility, but are deprecated and run spark-2. More information about Spark models can be found in the Agent documentation.
• Crawl an entire website and retrieve the content of all pages: returns a job ID. Note: the SDK automatically handles polling, providing a better developer experience.
• Instantly discover all URLs on a website.
• Find a specific URL within a site.
• Crawl multiple URLs at once.
• The SDK provides a convenient way to use all Firecrawl features and automatically handles polling for asynchronous operations.

Installation methods:

• Install the SDK: installation commands are provided for multiple languages or platforms (specific commands are not listed in detail in the material; only installing the SDK, adding dependencies (Gradle/Maven), and adding dependencies are mentioned).
• Agents and AI tools: Firecrawl skills directory—install using npx skills add firecrawl/skills. Build skills (integrating Firecrawl into product code) are written under skills/ in this repository and mirrored into the directory by CI. Contributing skills: CLI skills (including research/developer index skills) → PR firecrawl/cli; build/SDK skills → PR this repository (skills/); workflow skills → PR firecrawl/firecrawl-workflows. The directory (firecrawl/skills) is read-only—never PR directly.
• View all integrations.

Use cases:

• Provide real-time web data for AI agents.
• Build AI applications that need dynamic web content.
• Scenarios that require extracting structured data or Markdown from websites.
• Handle JS-heavy pages, PDFs, DOCX, etc.
• Integrate with various AI tools and MCP clients.

Limitations:

• The open-source version uses the AGPL-3.0 license, and the SDK and some UI components use the MIT license. The cloud version (firecrawl.dev) includes additional features.
• For local running, see the contribution guide; for self-hosting, see the self-hosting guide.
• End users are responsible for complying with website policies when crawling, and are advised to comply with applicable privacy policies and terms of use. By default, Firecrawl respects robots.txt directives. By using Firecrawl, you agree to comply with these conditions.
• Specific fees, pricing, and other information are not provided in the material.