SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI agentsintercept-mcp

intercept-mcp

Fetch web content as clean Markdown through a multi-tier fallback chain, handling tweets, YouTube, arXiv, PDFs and regular pages.

Open website

About this tool

intercept-mcp is an MCP server for AI-oriented web content scraping, with the goal of converting web content into clean Markdown that AI can directly use, rather than raw HTML or error pages. It can be used with basic functionality without an API key, and with a single command npx -y intercept-mcp it can run in any MCP client that supports stdio, including Claude Code, Claude Desktop, Codex, Cursor, Windsurf, Cline, and others.

Purpose

To provide AI agents with reliable web reading capabilities, solving the common problems of 403 errors, anti-scraping blocks, or returning large amounts of raw HTML when directly accessing URLs.

Main capabilities

  • Multi-level fallback chain: URLs pass in sequence through site-specific handlers, the shared cache, and the fallback pipeline. If the first strategy fails, up to 14 additional strategies are tried before giving up.
  • Site-specific handlers: routes to dedicated handlers for specific URL patterns, covering tweets, YouTube videos (with subtitles when available), arXiv papers, PDFs, Wikipedia articles, and GitHub repositories. GitHub API endpoints can work unauthenticated (60 requests per hour); setting GITHUB_TOKEN can increase the limit.
  • Shared cache: each request first checks agentsweb.org, a globally shared Markdown cache for AI agents, supported by a 9-source parallel scraping pipeline and JS/SPA rendering support. If another agent has already scraped the URL, results can be returned within 50ms. Each successful scrape automatically contributes back. You can fully opt out via INTERCEPT_SHARED_CACHE=false, or use read-only mode via INTERCEPT_CACHE_READ_ONLY=true.
  • Fallback pipeline: when no handler matches or a handler returns nothing, the URL enters a multi-level pipeline. Tier 2 scrapers run in parallel, and when multiple succeed, the highest-quality result is selected; the remaining tiers execute sequentially. All scrapers return canonical Markdown via Turndown (headings, links, bold, tables, code blocks).
  • In-memory cache: results are cached by TTL (60 minutes for success, 5 minutes for failure), up to 250 entries, with LRU eviction. Failed URLs are also cached to prevent retries of known invalid addresses. Three parameters can be configured via INTERCEPT_CACHE_TTL_MS, INTERCEPT_CACHE_FAILURE_TTL_MS, and INTERCEPT_CACHE_SIZE.
  • Structured output: reports source, quality, contentLength, truncated, nextStartIndex, cacheAgeSeconds, so the agent can branch accordingly.
  • Image support: direct image URLs (.png, .jpg, .gif, .webp, max 5 MB) are returned as MCP image blocks rather than text, letting the agent's own vision model read charts, diagrams, screenshots, and scanned documents.

Tools

  • fetch: Fetches a URL and returns clean Markdown. Parameters url (required), maxTier (1-5, optional), maxLength (default 50000), startIndex (default 0, for paginating long content), noCache (optional, skips cache for a live fetch). Long pages are truncated at maxLength and the startIndex to continue is provided.
  • fetch_batch: Fetches up to 10 URLs in parallel, each going through the same processor/fallback chain. Parameters urls (1-10, required), maxTier, noCache, maxLength (default 20000, budget per URL).
  • search_and_fetch: Searches the web and fetches the top results in a single call. Parameters query (required), count (1-5, default 3), maxLength (default 20000), site (optional), freshness (day/week/month/year).
  • search: Searches the web and returns results. Parameters count (1-20, default 5), site, page (1-10). Prefers the Brave Search API (if BRAVE_API_KEY is set), then SearXNG (if SEARXNG_URL is set), and finally DuckDuckGo (unstable; freshness and page are ignored).
  • extract: Extracts specific values from a page as JSON rather than Markdown prose. Parameters url (required), selectors (a mapping of field names to CSS selectors, or { selector, attr?, all? }), tables (boolean, defaults to true when no selectors are given).
  • Resource intercept://session/recent: a Markdown list of URLs fetched and cached in this session, most recent first, re-fetches return instantly.
  • Skills: research (searches a topic and fetches the top results for a multi-source summary; parameters topic, depth default 3); extract-article (fetches a URL and extracts key points; parameter url).

Environment variables and configuration

  • Search: the DuckDuckGo fallback is rate-limited and unstable. For production, it is recommended to self-host SearXNG and set SEARXNG_URL (the repository includes a Docker configuration; after running it, set SEARXNG_URL=http://localhost:8888，无速率限制、无 CAPTCHA, aggregating Google, Bing, DuckDuckGo, Wikipedia, Brave), or obtain a Brave Search API key.
  • Fetching: works without an API key. Setting CF_API_TOKEN + CF_ACCOUNT_ID enables Cloudflare Browser Run to handle JavaScript-heavy pages (SPAs, React sites).
  • Stealth fetching (USE_STEALTH_FETCH): disabled by default, must be explicitly enabled. When enabled, it adds a fetcher that uses got-scraping to mimic a real browser TLS fingerprint (Chrome/Firefox cipher suites, HTTP/2 settings, header order), which can bypass bot detection and CAPTCHA triggers on some sites. It runs at tier 3 and retries when the raw fetch is blocked. It may violate the terms of service of some websites; the author assumes no responsibility for its use.
  • Challenge solving (FLARESOLVERR_URL): stealth fetching cannot execute JavaScript challenges, and Cloudflare checks / DDoS-Guard interstitial pages will still block it. You can run FlareSolverr (Docker) and set FLARESOLVERR_URL=http://localhost:8191。在 tier 3 runs as a last resort, enabled only when this variable is set, solving may take 30-60 seconds, and it is the slowest scraper.
  • Commercial web unlocker (WEB_UNLOCKER_URL): for the hardest targets requiring residential IP rotation, real browser rendering, and CAPTCHA handling, it supports any unlocker that exposes a "GET URL returns HTML" endpoint, configured with an API key via a template containing the {url} placeholder. It runs as a paid last resort in tier 3, enabled only when this variable is set, with credentials sent only to the unlocker. Billed per request.
  • Bring your own proxy (HTTPS_PROXY): follows the standard HTTPS_PROXY / HTTP_PROXY / NO_PROXY environment variables, and can route all outbound traffic. It works with any HTTP(S) proxy, and stealth scraping and got-scraping calls also use it automatically.
  • Proxy rotation (INTERCEPT_PROXIES): set to a comma- or space-separated list; intercept-mcp rotates through them, and on a block (HTTP 403, 429, 451, 503) or error automatically retries through the next proxy (up to 3 times). It takes precedence over HTTPS_PROXY, is applied per request, accepts HTTP(S) proxies, and ignores invalid entries.
  • Per-domain authentication (INTERCEPT_AUTH): a JSON object mapping domains to header maps, allowing you to attach your own headers or cookies to read legitimately logged-in content. Security model: credentials are sent only to the configured origin; when scraping through Jina, web archives, CORS proxies, FlareSolverr, or a shared cache, the intermediary connects to a different host, so cookies/tokens are not sent to them. Authenticated responses do not enter the shared cache (no reads or writes to agentsweb.org), while the session cache still applies. Treat values as secrets; environment variables are visible to the process and child processes and may be captured by shell history or process lists. Provide credentials only for accounts you own or are authorized to use.

Other features

  • URL normalization: automatically cleans incoming URLs, stripping 60+ tracking parameters (UTM, click IDs, analytics, A/B testing, etc.), removing hash fragments, upgrading to HTTPS, cleaning up AMP traces, and preserving functional parameters (ref, format, page, offset, limit).
  • SSRF protection: rejects URLs pointing to local or internal basefacilities, including loopback and private IPv4/IPv6 ranges, link-local addresses (including the 169.254.169.254 cloud metadata endpoint), CGNAT, multicast/reserved ranges, and local hostnames (localhost, *.local, *.internal, *.home.arpa). It checks literal IPs, including alternative notations normalized by URL parsers (decimal, hexadecimal); it does not resolve DNS, so public hostnames pointing to private IPs will not be caught.
  • Content quality detection: every scraper result is scored. The following are automatically judged as failures: CAPTCHA / Cloudflare challenge, HTTP error page in the body, content under 200 characters.

Use cases

MCP client environments that need AI agents to reliably read web pages, papers, video subtitles, PDFs, GitHub repositories, Wikipedia, and other content; scenarios that need to search and scrape multi-source content in a single call; scenarios that need to bypass light anti-scraping or JS-rendered pages.

Limitations

  • Basic scraping requires no API key, but the DuckDuckGo fallback for search is unstable and rate-limited.
  • Stealth scraping may violate the terms of service of some websites, is disabled by default, and is used at your own risk.
  • FlareSolverr may take 30-60 seconds to solve a challenge and is the slowest option.
  • Commercial unlockers are billed per request.
  • It does not resolve DNS and cannot catch public hostnames pointing to private IPs.
  • Cookies in INTERCEPT_AUTH expire and need to be refreshed periodically.

Related tools

Browse category