PyreCrawl is a self-hosted web browsing tool for AI agents, provided as an MCP (Model Context Protocol) server. It integrates capabilities such as scrape, extract, crawl, map, search, batch, research, and monitor into a single tool interface for MCP-compatible agents to call. The project is positioned as a free alternative to Firecrawl, emphasizing self-hosting, no API key required, no rate limits, and no subscription.
Main capabilities and tools: PyreCrawl exposes multiple tools through MCP, supporting operations such as scrape, crawl, extract, map, search, batch, research, and monitor. It also provides MCP resources (read-only status, no tool call required): pyrecrawl://cache/stats, pyrecrawl://sessions, pyrecrawl://monitors; as well as MCP prompts (ready-made playbooks): research(topic), rag_ingest(site), watch_page(url). Tools provide a prefer option: "auto" (default ladder), "fast" (HTTP only), "stealth" (Cloudflare bypass), "llm" (deep processing).
Intelligent fallback ladder: PyreCrawl runs three tiers for each request and stops at the first tier that returns complete, LLM-ready results, so agents do not need to choose by themselves; prefer="auto" does this automatically on every call.
Dynamic DOM data handling: Some websites keep the required data in DOM properties (for example, the .value of an <input>), which is written by JS after an XHR and does not appear in the serialized HTML. For this, the scrape tool accepts two stealth-tier parameters: wait_for (a JS predicate expression that is polled until true, limited by timeout, used to handle asynchronously arriving content and avoid guessing sleep times); js (a JS expression evaluated once after the page stabilizes, with the return value returned through meta.js_result; errors are recorded in meta.js_error, and the page result is still returned without crashing).
Installation and usage: Using UV (a fast Python package manager that can handle Python itself, with no separate Python installation required) provides a one-time installation experience. First install UV, then run PyreCrawl directly (no venv, no pip install, no Python download). Then perform the one-time browser engine installation, installing Chromium + stealth browser engine (about 2 minutes, one-time). After that, register PyreCrawl with the AI agent. Supported agents include claude-desktop, claude-code, cursor, vscode, codex, opencode, hermes. After installation and registration, restart the agent (or start a new session), and you can ask questions in natural language, for example, "Scrape https://example.com and summarize it.”, no special syntax required.
Manual configuration: If pyrecrawl install does not match the user's environment, you can configure it manually. Claude Desktop configuration file paths: on Linux, ~/.config/Claude/claude_desktop_config.json; on macOS, ~/Library/Application Support/Claude/claude_desktop_config.json; on Windows, %AppData%\Claude\claude_desktop_config.json. Claude Code uses a project-level .mcp.json. Cursor uses ~/.cursor/mcp.json. VS Code / Copilot uses a project-level .vscode/mcp.json. Codex uses ~/.codex/config.toml. OpenCode uses ~/.config/opencode/opencode.json. Hermes uses ~/.hermes/config.yaml on Linux/macOS and %LocalAppData%\hermes\config.yaml on Windows. Windows note: uvx must be in PATH; otherwise use the full path to uvx.exe (for example, C:\Users<you>\AppData\Local\hermes\bin\uvx.exe).
Skills: PyreCrawl tools give the agent "hands" (scraping, crawling, searching), but the agent still needs a "brain"—instructions about when to use which tool, how to chain together the research process, and which anti-hallucination rules to follow. This is exactly what PyreCrawl Skills provide. Without skills, the agent has powerful tools but uses them improvised; with skills, the agent follows proven research protocols and anti-hallucination guardrails.
Comparison with hosted Firecrawl: The project provides a comparison with hosted Firecrawl (the specific comparison details are not fully expanded in the given materials).
Release and updates: Only maintainers can publish to PyPI (built via GitHub Actions and uploaded using trusted publishing). PyreCrawl checks PyPI on every startup and reports the latest version; MCP agents can automatically see this information through the health() tool response and can notify the user inline. You can also check manually. To receive new version notifications, you can click Watch → Releases only in the GitHub repository so that you receive email notifications when a new version is released.
Privacy: PyreCrawl sends an anonymous usage ping once every 24 hours when the MCP server starts, in order to count real users (DAU/MAU) rather than raw download counts. The client code is located at src/pyrecrawl/telemetry.py (about 90 lines, using only the standard library); the collector is located at workers/telemetry/, which is a self-hostable Cloudflare Worker + D1 and does not use third-party analytics services. You can opt out at any time.
License: MIT, see LICENSE for details.
Use cases: Self-hosted environments that need to provide web scraping, crawling, searching, research, and monitoring capabilities for AI agents, especially for users who want to avoid API keys, rate limits, and subscription costs. Limitations: Specific pricing, prices, performance metrics, supported website scope, and other information are not yet fully provided in the given materials.
