SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsfetcher-mcp

fetcher-mcp

Fetch web page content using a Playwright headless browser, supporting JavaScript rendering, content extraction and Markdown or HTML output.

Open website

About this tool

fetcher-mcp is an MCP (Model Context Protocol) server that fetches web page content through the Playwright headless browser. It is mainly intended for users who need to integrate web scraping capabilities into AI clients or automated workflows.

Purpose

This tool is used to scrape web page content, and is especially suitable for handling dynamic pages and modern web applications that rely on JavaScript rendering. Unlike traditional web scrapers, it uses Playwright to execute JavaScript, so it can retrieve dynamically generated content.

Main capabilities

  • JavaScript support: Uses Playwright to execute JavaScript and handle dynamic web content.
  • Intelligent content extraction: Built-in Readability algorithm automatically extracts the main body of a web page and removes unnecessary elements such as ads and navigation.
  • Flexible output formats: Supports both HTML and Markdown output formats, making it easy to integrate into downstream applications.
  • Parallel processing: The fetch_urls tool supports concurrent scraping of multiple URLs, improving the efficiency of batch operations.
  • Resource optimization: Automatically blocks unnecessary resources (images, stylesheets, fonts, media), reducing bandwidth usage and improving performance.
  • Robust error handling: Provides comprehensive error handling and logging.
  • Configurable parameters: Allows fine-grained control over timeouts, content extraction, and output format.

Usage

Run directly via npx

For initial setup, install the required browser by running the corresponding command in the terminal.

HTTP and SSE transports

Using the --transport=http parameter starts both the Streamable HTTP endpoint and the SSE endpoint. After startup, the server provides the following endpoints:

  • /mcp - Streamable HTTP endpoint (modern MCP protocol)
  • /sse - SSE endpoint (legacy MCP protocol)

Clients can choose the connection method as needed.

Use the --debug option to show the browser window for debugging.

Configure MCP

Configure this MCP server in Claude Desktop:

  • MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
  • Windows: %APPDATA%/Claude/claude_desktop_config.json

Docker deployment

Running with Docker is supported, and deployment with Docker Compose is also supported (a docker-compose.yml file must be created).

Tools and parameters

fetch_url

Fetches web page content from the specified URL. Uses the Playwright headless browser to parse JavaScript, supports intelligent extraction of the main body, and converts it to Markdown. Parameters include:

  • url: The web page URL to scrape (required)
  • timeout: Page load timeout (milliseconds), default 30000 (30 seconds)
  • waitUntil: Navigation completion timing, options are 'load', 'domcontentloaded', 'networkidle', 'commit', default 'load'
  • extractContent: Whether to intelligently extract the main body, default true
  • maxLength: Maximum length of returned content (characters), default unlimited
  • returnHtml: Whether to return HTML instead of Markdown, default false
  • waitForNavigation: Whether to wait for additional navigation after the initial page load (suitable for sites with anti-bot verification), default false
  • navigationTimeout: Maximum time to wait for additional navigation (milliseconds), default 10000 (10 seconds)
  • disableMedia: Whether to disable media resources (images, stylesheets, fonts, media), default true
  • debug: Whether to enable debug mode (show the browser window); if specified, overrides the --debug command-line flag

fetch_urls

Fetches web page content from multiple URLs in parallel. Uses multiple tabs for parallel scraping to improve performance, and returns merged results with clear separation between each web page. Parameters include: urls (the array of URLs to scrape, required), and the other parameters are the same as fetch_url.

browser_install

Automatically installs the Playwright Chromium browser binary. Installs the required Chromium binary when the browser is unavailable, and automatically suggests when a browser installation error occurs. Parameters include: withDeps (installs the system dependencies required for Chromium, default false), force (forces installation even if Chromium is already installed, default false).

Special Website Scenario Handling

  • Wait for full loading: For websites that use CAPTCHAs, redirects, or other verification mechanisms, you can add "please wait for the page to fully load" to the prompt, which will use the waitForNavigation: true parameter.
  • Increase timeout: For slow-loading websites, you can set the page load timeout to 60 seconds, adjusting the timeout and navigationTimeout parameters accordingly.
  • Preserve the original HTML structure: When content extraction may fail, you can preserve the original HTML content by setting extractContent: false and returnHtml: true.
  • Scrape the full page content: When too little content is extracted, you can scrape the full web page content instead of just the main text by setting extractContent: false.
  • Return HTML format: When HTML format is needed instead of the default Markdown, set returnHtml: true.

Debugging and Authentication

  • Activate dynamic debugging: Display the browser window during a specific scraping operation by setting debug: true, even if --debug was not used when the server started.
  • Manual login: When logging in with your own credentials, you can run in debug mode by setting debug: true or using the --debug flag, keeping the browser window open to log in manually.
  • Interact with the debug browser: After enabling debug mode, the browser window remains open, allowing you to manually log in to the website. After logging in, content will be scraped using the authenticated session.
  • Enable debugging for a specific request: Even if the server is already running, you can enable debug mode for a specific request by setting debug: true to open the browser window for that request only for manual login.

Installation and Build

  • Install dependencies
  • Install the Playwright browser
  • Build the server
  • Debug with MCP Inspector
  • You can also enable visible browser mode for debugging

Related Projects

g-search-mcp: An MCP server for Google search that supports parallel multi-keyword search, suitable for batch searching and data collection.

This project uses the MIT License.

Limitations

The materials do not provide specific performance metrics, scraping success rates, a complete list of compatible MCP clients, or pricing information.

Related tools

Browse category