Agent Scraper MCP Server is a web scraping service for AI agents, providing web content retrieval and structured extraction capabilities through MCP (Model Context Protocol) and a REST API. It is positioned as "one of the most frequently requested practical tools for AI agents," focusing on professional web scraping, screenshots, and content extraction.
Main capabilities The toolset includes six functions in total:
- Cleaned content extraction (scrape_url): extracts Readability-like readable text from any web page, with output formats supporting text, markdown, or html, defaulting to markdown. The returned content includes success status, URL, title, and body text.
- Structured scraping (scrape_structured): uses CSS selectors to extract specified data according to a "name → selector" dictionary, returning a structured result containing data.
- Screenshot (screenshot_url): uses Playwright to capture web page screenshots, with configurable viewport width (default 1280), height (default 720), and whether to capture the full page (default false), returning a base64-encoded PNG image.
- Link extraction (extract_links): retrieves all links on the page, with optional regular expression filtering of URLs, returning an array of links containing text and href plus the count.
- Metadata extraction (extract_meta): extracts the page title, meta description, canonical URL, favicon, Open Graph tags, and Twitter Card tags.
- Google search (search_google): programmatically retrieves Google results, with a configurable number of results to return (default 10), returning results as an array containing titles, URLs, and snippet fragments.
Usage You can add the service to an MCP configuration file (for example cline_mcp_settings.json or a similar file); the base URL is https://agent-scraper-mcp.onrender.com。所有工具均已包含在服务中。也可直接调用 REST API, and the documentation address is https://agent-scraper-mcp.onrender.com/docs。
Pricing and quota The free quota is 50 requests per IP per day, with no credit card required. After exceeding the free quota, charges apply per request: scraping tools (scrape_url, scrape_structured, extract_links, extract_meta, search_google) cost $0.005 per request; the screenshot tool costs $0.01 per request due to higher computational cost. Payment uses the x402 protocol (HTTP 402), using USDC on the Base chain, and the receiving wallet address is 0x8E844a7De89d7CfBFe9B4453E65935A22F146aBB. When the quota is exceeded, the server returns HTTP 402 and a machine-readable payment requirement (x402 v1); clients that support x402 can automatically pay and retry. Manual payment requires attaching an X-Payment header containing a base64-encoded JSON envelope, which includes an EIP-3009 TransferWithAuthorization signed with the payer's EIP-712 key (USDC on Base), with an exact amount, the receiving address set to the above wallet, and an unused nonce; the server performs signature recovery and validation of the amount, recipient, deadline, replay, and so on before processing the request. This mechanism applies to both the REST API and MCP tools/call.
Server operator receiving payments If X402_SETTLER_PRIVATE_KEY is set, each verified payment is executed on-chain within seconds: submitting an EIP-3009 transferWithAuthorization to the USDC contract on Base, with USDC transferred from the payer to X402_WALLET_ADDRESS. The settlement key does not need to be the key of the receiving wallet; any address can execute this authorization, so a dedicated temporary gas-only key can be used to submit the transaction, while the funds still go to the main wallet; that key only needs to hold a small amount of ETH on Base for gas. Other environment variables include X402_RPC_URL (default https://mainnet.base.org）与 X402_SETTLE_MIN_MICROUSD (default 0, meaning each payment is settled immediately; increasing this value enables batch settlement of small payments). Failed settlements are retried according to a backoff strategy. If all retries fail, the complete authorization information is written to the logs at CRITICAL level for manual redemption, so payments will not be silently lost.
Tech Stack and Deployment The project uses Python 3.11 with modern async/await, adopts FastAPI as the REST framework, FastMCP to implement the MCP protocol (Streamable HTTP), Playwright for browser automation screenshots, httpx as the asynchronous HTTP client, BeautifulSoup4 for parsing HTML, and readability-lxml for content extraction (similar to Firefox Reader View). The license is MIT. It is deployed on the Render free tier, with the service name agent-scraper-mcp, runtime Docker, and supports automatic deployment from the GitHub repository aparajithn/agent-scraper-mcp; environment variables include PUBLIC_HOST and X402_WALLET_ADDRESS.
Applicable Scenarios and Limitations This service is suitable for AI agent workflows that require web page body cleaning, extracting structured data by selector, web page screenshots, link and metadata collection, and programmatic Google search. In terms of limitations: the free tier allows 50 requests per IP per day, and excess usage requires payment in USDC at the rates above; the screenshot tool costs more than other tools; the materials do not provide specific concurrency or timeout limits beyond the rate cap, the range of supported data formats, or whether login state or anti-scraping bypass is supported; this information is not yet available.
