webcrawl-mcp is a lightweight MCP (Model Context Protocol) server designed to give Claude Code or any MCP-compatible client the ability to scrape, search, map, and crawl web pages. It is built entirely on free and open-source libraries, with Firecrawl serving only as an optional fallback for handling sites with heavy JavaScript rendering (you need to provide your own API Key).
Design Philosophy
Most web scraping does not actually require a headless browser. trafilatura can handle about 80% of common scenarios locally (articles, documentation, blogs), is faster, and minimizes the use of external APIs. This server delegates simple content to local extraction and only falls back to Firecrawl when the content quality is genuinely poor.
Source Identification
The source field on scraped content may have the values static_http, static_http_retry, firecrawl_transport_fallback, or firecrawl_quality_fallback, used to indicate which path produced the content.
Environment Requirements
Requires Python 3.12 or later.
Quick Smoke Test
After running, it should print "Webcrawl MCP server running", then exit normally with Ctrl-C.
Installing from Source (for Development)
Install and develop via a source checkout.
Configuring the MCP Client
Create .mcp.json in the project root directory (or merge it into ~/.claude/settings.json). This approach uses uvx to run the package in a temporary environment, with no manual installation required. If uvx is unavailable, you can install with pip install webcrawl-mcp and set "command" to "webcrawl-mcp" with no arguments.
If using a source checkout (for development), point command to the Python in the virtual environment and use "args": ["-m", "webcrawl_mcp"].
Then run /mcp in a Claude Code session, and you should see webcrawl listed, andwith four tools.
Claude Desktop uses the same JSON structure, placed in claude_desktop_config.json (see the Anthropic documentation for paths on each operating system), and restart Claude Desktop after editing.
Other MCP clients (Cursor, Cline, Continue, Zed, etc.) use the standardized command / args / env structure; consult the corresponding client's MCP documentation for the specific placement location.
Verifying That It Works
You can ask your agent to perform something like: Use the webcrawl_scrape tool to scrape https://docs.python.org/3/library/asyncio.html and summarize the content of section one. You should see the tool call in the client UI, followed by a summary based on the live page content. If there is no response, see the troubleshooting section.
Environment Variables
Set these in the env block of the MCP configuration, not in the shell—the MCP server runs under the client, not in your terminal.
Fallback Behavior
The crawler distinguishes between extraction quality failures and transport failures, and reports via the source field which path produced the content.
Extraction quality path (default): trafilatura extracts the main content from HTML → source: static_http. If it fails or returns fewer than 200 characters, markdownify converts the raw HTML. If the result is still of poor quality and FIRECRAWL_API_KEY is set, Firecrawl is used as a last resort → source: firecrawl_quality_fallback.
Transport path (must be explicitly enabled): If the request returns 403, 429, or 503 (typical anti-crawling responses): Under POLITE_MODE=true (default), a 429 with a Retry-After header gets one bounded retry of the original request → on success, source: static_http_retry. When FALLBACK_ON_TRANSPORT_ERROR=true and FIRECRAWL_API_KEY is set, the request goes through Firecrawl instead of throwing an error → source: firecrawl_transport_fallback. Otherwise, the transport error is thrown to the caller (current behavior).
Without a Firecrawl key, the tool is fully self-contained and free; without a key, FALLBACK_ON_TRANSPORT_ERROR has no effect.
Troubleshooting
The client does not list webcrawl in the MCP server: this is almost always a command path issue; it must be the absolute path to the Python binary in the virtual environment, not simply python. You can test in the terminal: the path you configured with -m webcrawl_mcp should print Webcrawl MCP server running. ModuleNotFoundError: webcrawl_mcp: this may be because pip install -e . was executed in a different virtual environment from command, or PYTHONPATH is missing/incorrect. Please confirm that both point to the same checkout. Python version mismatch: 3.12+ is required; python --version inside the virtual environment should report ≥ 3.12, otherwise rebuild the virtual environment with a newer version of Python. Crawling returns very little text: some sites use JavaScript rendering and cannot be extracted statically. You can set FIRECRAWL_API_KEY to enable the fallback path, or accept that the tool is not suitable for that particular site. Search is slow or rate-limited: DuckDuckGo rate-limits bursty queries. Please lengthen the interval between searches, or reduce num_results.
Responsible Use
This tool is intended for obtaining public web content for research, coding assistance, and similar legitimate uses. You are responsible for: complying with the target site's terms of service and robots.txt; not overloading servers (the built-in per-domain rate limiting helps, but do not bypass it); and complying with applicable laws in your jurisdiction regarding automated access and data use.
License
MIT — see LICENSE.
Acknowledgments
Built on top of trafilatura, httpx, beautifulsoup4, markdownify, ddgs, and fastmcp. The Firecrawl integration uses the public Firecrawl API (no affiliation).
