Purpose
mcp-read-website-fast is used to provide fast, low-token web content extraction for AI agents, Claude Code, IDEs, and LLM pipelines. It converts web pages into clean Markdown and preserves links so that large language models can obtain useful content without parsing the entire page HTML.
Key Capabilities
- Fast startup: Uses the official MCP SDK and lazy loading to optimize performance.
- Content extraction: Uses Mozilla Readability, the same as Firefox Reader View.
- HTML to Markdown: Uses Turndown with GFM support.
- Smart caching: Based on SHA-256 hashed URLs.
- Polite crawling: Supports robots.txt and rate limiting.
- Concurrent fetching: Supports crawling with configurable depth.
- Streaming-first design: Reduces memory usage.
- Link preservation: Suitable for knowledge graphs.
- Optional chunking: Facilitates downstream processing.
Note: This package now uses @just-every/crawl as the core crawling and Markdown conversion functionality.
Usage
This tool is integrated as an MCP server into MCP-compatible clients. In Settings → Tools → AI Assistant → Model Context Protocol (MCP) → Add, select "As JSON" and paste the configuration; or enter /add in the chat window and fill in the same JSON.
The raw JSON can be used with any MCP client by placing it in the client's mcp.json, such as .vscode/mcp.json, ~/.cursor/mcp.json, or Claude's .mcp.json.
Available Tools
read_website - Fetches a web page and converts it into clean Markdown. Parameters:
- url (required): The HTTP/HTTPS URL to fetch.
- pages (optional): Maximum number of pages to crawl, default 1, maximum 100.
Available Resources
- read-website-fast://status - Get cache statistics.
- read-website-fast://clear-cache - Clear the cache directory.
Development Usage
Single-page fetching. With depth crawling, the available parameters include:
- -p, --pages <number>: Maximum number of pages to crawl, default 1.
- -c, --concurrency <number>: Maximum number of concurrent requests, default 3.
- --no-robots: Ignore robots.txt.
- --all-origins: Allow cross-origin crawling.
- -u, --user-agent <string>: Custom user agent.
- --cache-dir <path>: Cache directory, default .cache.
- -t, --timeout <ms>: Request timeout in milliseconds, default 30000.
- -o, --output <format>: Output format, options are json, markdown, or both, default markdown.
Auto-Restart Feature
The MCP server includes auto-restart capability by default to improve reliability:
- Automatically restarts when the server crashes.
- Handles uncaught exceptions and Promise rejections.
- Implements exponential backoff (up to 10 attempts within 1 minute).
- Logs all restart attempts for monitoring.
- Gracefully handles shutdown signals (SIGINT, SIGTERM).
To disable auto-restart during development/debugging, refer to the project documentation.
Use Cases
- Extract web page body content for AI agents, reducing irrelevant content and token consumption.
- Quickly read web pages and convert them to Markdown in an IDE or Claude Code.
- LLM pipelines that need to preserve links to build knowledge graphs.
- Need local, low-dependency, polite concurrent crawling of websites.
Limitations
- Some websites block automated access.
- You can try using the -u parameter to set a custom user agent.
- Websites that require JavaScript are not supported.
- If you encounter timeouts, you can use the -t parameter to increase the timeout.
- Check the network connection and confirm that the URL is accessible.
Contributing
Contributions are welcome: Fork the repository, create a feature branch, add tests for new features, and submit a pull request.
