Purpose
ai-first-scraper-mcp is an MCP (Model Context Protocol) server designed to let MCP-compatible agents such as Claude Desktop, Cursor, and Cline directly access ad-free web scraping and search capabilities. It solves the problem that agents face when calling curl themselves and then struggling to parse large amounts of ad-laden HTML—what the agent receives is clean Markdown that can be used directly for reasoning.
Main Capabilities
This server adds three tools to any MCP-compatible agent:
- fetch_page: scrape a single page.
- fetch_pages_batch: scrape multiple pages in parallel in batches.
- search_web: search the web and return the top k pages. All scraping and search results are powered by the ai-first-scraper and ai-first-search APIs, with output in clean Markdown.
How to Use
Integration is simple and requires only one line of configuration. uvx is recommended (no installation required, runs on demand from PyPI, fastest); pip install is also an option. After configuration is complete, restart the client (Claude Desktop / Cursor / Cline), and the three tools will appear automatically. The location of the configuration file is given in the project documentation. By default, the server calls the public ai-first-scraper.onrender.com and ai-first-search.onrender.com instances. For self-hosting, you can set environment variables in the MCP configuration to point to your own backend. Verification method: In the MCP client, have the agent execute an instruction such as "use the search_web tool to find the 3 most recent articles about MCP, and summarize each in 5 key points." The agent should call search_web, obtain the Markdown for each article, and generate summaries without touching the raw HTML.
Applicable Scenarios
Suitable for scenarios that require AI agents to read web page content, perform web searches, and obtain structured text, such as information retrieval, content summarization, and research assistance.
Limitations
The materials do not provide specific limitation information about scraping frequency, API usage quotas, pricing models, or supported platforms. This server relies on the ai-first-scraper and ai-first-search APIs, uses public instances by default, and can also be self-hosted, but the specific requirements for self-hosting are not explained in the materials. The project uses the MIT license.
