SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsanycrawl-mcp-server

anycrawl-mcp-server

AnyCrawl MCP Server, Powerful web scraping and crawling for Cursor, Claude, and other LLM clients via the Model Context Protocol (MCP).

Open website

About this tool

AnyCrawl MCP Server is a server that provides web scraping and crawling capabilities for LLM clients through the Model Context Protocol (MCP). The project homepage is https://github.com/any4ai/anycrawl-mcp-server。它将网页数据转换为适合大语言模型消费的格式，并支持从搜索引擎获取结构化 SERP results.

[Purpose] It is mainly used to enable LLM clients such as Cursor, Claude, VS Code, Windsurf, and Codex to scrape a single web page, crawl an entire site's content, and search the web and scrape the results.

[Main Capabilities]

  1. Web scraping (Scrape): Extract content from a single URL, with support for multiple output formats.
  2. Full-site crawling (Crawl): Crawl multiple pages with configurable depth and page limits, waiting for completion by default and returning aggregated results.
  3. Search engine integration (Search/SERP): Search the web and optionally scrape the search results.
  4. Multiple engines: auto (intelligent selection), Playwright, Cheerio, Puppeteer.
  5. Flexible output: Markdown, HTML, text, screenshot, screenshot@fullPage, rawHtml, json (structured data extraction).
  6. Asynchronous operations: Non-blocking crawl tasks with status monitoring support.
  7. Error handling and logging: Parameter validation errors, API errors, network errors, automatic retry and backoff for rate limits, and multi-level logging.
  8. Multiple modes: STDIO (default), MCP(HTTP), SSE; when deployed with Docker Compose, MCP (port 3000) and SSE (port 3001) run in the same container, fronted by Nginx, and support the /{API_KEY}/mcp and /{API_KEY}/sse paths.

[Provided Tools]

  • anycrawl_scrape: Scrape a single URL. Parameters include url (required), engine (auto/playwright/cheerio/puppeteer, default auto), formats (markdown, html, text, screenshot, screenshot@fullPage, rawHtml, json), proxy, timeout (default 300000 milliseconds), retry (default false), wait_for, include_tags, exclude_tags, json_options.
  • anycrawl_crawl: Start a crawl task to scrape multiple pages. By default, wait for completion and return aggregated results (poll every 3 seconds, 60-second timeout). Parameters include url (required), max_depth (default 10), limit (default 100), strategy (all, same-domain, same-hostname, same-origin), exclude_paths, include_paths, scrape_options, poll_seconds (default 3), timeout_ms (default 60000). Returns a form like { "job_id": "...", "status": "completed", "total": N, "completed": N, "creditsUsed": N, "data": [...] }.
  • anycrawl_crawl_status: Query crawl task status, parameter job_id (required).
  • anycrawl_crawl_results: Get crawl task results, parameters include job_id (required), skip (pagination).
  • anycrawl_cancel_crawl: Cancel a pending crawl task, parameter job_id (required).
  • anycrawl_search: Use the AnyCrawl search engine to search the web. Parameters include query (required), engine (google), limit (default 10), offset (default 0), pages, lang, country, scrape_options (required), safeSearch (0 off, 1 moderate, 2 strict).

[Usage] Two deployment modes are supported:

  1. Cloud service (recommended): Use https://mcp.anycrawl.dev，无需自建服务器。云端端点：MCP（streamable_http）为 https://mcp.anycrawl.dev/{API_KEY}/mcp（适用于 Codex and Cursor); SSE is https://mcp.anycrawl.dev/{API_KEY}/sse（仅旧版 SSE clients, not suitable for Codex).
  2. Self-hosted: Configure the base URL to point to your own AnyCrawl API instance; for local development, host/port can be customized. Get an API Key: Visit the AnyCrawl website to register or log in (https://anycrawl.dev），注册可免费获得 1500 credits (approximately 1500 pages can be crawled); copy the key in dashboard → API Keys and set it as the environment variable ANYCRAWL_API_KEY.

Available modes: default STDIO (no environment variables required); set ANYCRAWL_MODE to switch. Docker image environment variables: ANYCRAWL_MODE (default MCP_AND_SSE in compose, or MCP, SSE), ANYCRAWL_MCP_PORT (default 3000), ANYCRAWL_SSE_PORT (default 3001), CLOUD_SERVICE (when true, extract the API Key from /{API_KEY}/... or the request header), ANYCRAWL_BASE_URL (default https://api.anycrawl.dev）。

Client configuration:

  • Cursor: requires v0.45.6+. In v0.48.6 and newer, the configuration can be added in the MCP Servers settings; for v0.45.6, go through Cursor Settings → Features → MCP Servers → "+ Add New MCP Server", set Name to "anycrawl-mcp" and Type to "command". The local and cloud configurations for HTTP mode (streamable_http) differ; when using streamable_http, Cursor does not need an API Key.
  • VS Code: for manual installation, add the corresponding JSON in User Settings (JSON) (Command Palette → Preferences: Open User Settings (JSON)); optionally place the configuration in .vscode/mcp.json for sharing.
  • Windsurf: add to ~/.codeium/windsurf/model_config.json.
  • Claude Desktop / Claude Code: add using the claude mcp add command; the Claude Desktop configuration file is ~/Library/Application Support/Claude/claude_desktop_config.json on macOS or %APPDATA%\Claude\claude_desktop_config.json on Windows.
  • SSE mode: intended for web applications, testing, and web-based LLM clients.
  • Codex: uses only streamable HTTP and does not support the legacy SSE. Write the configuration to ~/.codex/config.toml (replace YOUR_API_KEY), use the path /mcp instead of /sse (using /sse will cause Codex to hang until a timeout of about 30 seconds); put the API Key in the URL path (Codex will not read ANYCRAWL_API_KEY for remote servers); if startup still times out, you can increase startup_timeout_sec (Codex defaults to 10 seconds).

[Use cases]

  • Extract content from a single page, quick data extraction, test a specific URL (anycrawl_scrape).
  • Extract content from multiple related pages, full-site analysis, batch data collection (anycrawl_crawl).
  • Find specific information across multiple websites, research and discovery, when you are unsure which website has the target information (anycrawl_search).

[Limitations and requirements]

  • Requires Node.js 22.12+ (Docker and CI use Node 22, see .node-version).
  • The cloud service requires an API Key; self-hosting requires configuring a base URL pointing to a self-built AnyCrawl API instance.
  • The project uses the MIT License.
  • Feedback channels: GitHub Issues; documentation: AnyCrawl API Docs; email: help@anycrawl.dev.
  • Information such as specific pricing and support for other platforms is not provided.

Related tools

Browse category