Purpose
crawl-readiness-mcp is an MCP (Model Context Protocol) server for assessing a website's "AI readiness," i.e., whether more than 50 AI crawlers such as ChatGPT, Claude, Perplexity, and Google AI can access the website's content. It also provides fix generation capabilities, allowing llms.txt, robots.txt for AI crawlers, and JSON-LD structured data to be generated directly within an AI client, and reads LLM Monitor brand mention trends.
Main capabilities
The server provides eight tools that AI assistants can call, divided into three categories: auditing, generation, and monitoring.
Auditing (no registration required):
- check_ai_readiness: Scores a website from 0–100 against more than 50 AI crawlers and provides a prioritized fix list; additional items that do not affect the score are flagged.
- validate_schema: Audits JSON-LD structured data on any URL, supporting more than 20 types and type-specific rules.
- validate_robots: Audits robots.txt line by line.
- check_content_parity: Compares the content humans see with the content AI crawlers see.
Generation (requires a free API key):
- generate_llms_txt: Automatically crawls a website and generates a correctly formatted llms.txt.
- generate_robots_txt: Generates robots.txt for AI crawlers, with preset policies supported.
- generate_schema: Generates ready-to-paste Organization / WebSite / Article JSON-LD.
Monitoring (read-only, requires a free API key):
- get_monitor_trend: Checks whether ChatGPT, Claude, Perplexity, and Google AI mention your brand and how that compares with competitors, as well as week-by-week trends; reads your LLM Monitor project without triggering a run.
How to use
Claude Desktop one-click installation: Download crawl-readiness-mcp from the latest release-<version>.mcpb and double-click it (or drag it into the Claude Desktop window). Claude Desktop will show the extension features and ask for an optional API key. The bundle includes the server and its dependencies, so nothing else needs to be installed.
Claude Code plugin method: This repository is also a Claude plugin. The .mcp.json in the root directory starts the server (pinned to the current version), and the ai-readiness skill tells Claude when to use which tool and how to interpret the results. Once listed in the Claude directory, it can be added. Before that, you can clone the repository and load it to start Claude Code. When the plugin is enabled, Claude Code asks for an optional API key and stores it in secure storage; the plugin's own files do not contain the key.
Claude Desktop manual configuration: Open the configuration file (macOS: ~/Library/Application Support/Claude/claude_desktop_config.json; Windows: %APPDATA%\Claude\claude_desktop_config.json), add the configuration to the mcpServers block, restart Claude Desktop, and then "crawl-readiness" will be visible in the tools panel.
Cursor: Open Cursor Settings → Features → Model Context Protocol → Add new global MCP server.
Cline / Windsurf / Continue: In the same way, add crawl-readiness to the MCP server list, with the command npx -y crawl-readiness-mcp.
Unlock the generation tools: The four auditing tools work out of the box and require no registration. The three generation tools require an API key, which can be obtained for free at crawlreadiness.com/dashboard (the free tier includes 50 checks per month, no credit card required). After obtaining the key, add it via env in the same configuration, restart the client, and all eight tools can be called.
Example prompts: After installation, you can try them in an AI client, for example, "Is my portfolio siAI-readable? URL: yourdomain.com" "Check if ChatGPT can see acme-inc.com" "Validate the JSON-LD on stripe.com — is anything missing?" "Compare what humans see vs what AI crawlers see on producthunt.com" "Generate an llms.txt for my site and save it to /public/llms.txt" "My site scored 68/100 — walk me through the fixes". AI can automatically chain tools together: audit, identify issues, fix, and write to project files, with no manual copy-pasting.
What the tools return
Each tool returns the full JSON response of the Crawl Readiness API, the same data rendered by the web dashboard. For example, check_ai_readiness returns the score, robots.txt status for each crawler, structured data flags, homepage signals, and a prioritized fix list (fixes marked extra: true are optional and do not count toward the score); validate_schema returns issues for each block (errors, warnings, info) and suggested missing types; validate_robots returns line-by-line issues and an AI bot coverage summary; check_content_parity returns the verdict, HTTP status per crawler, word overlap percentage, and warnings; generate_llms_txt returns the file contents and an accompanying robots.txt snippet; generate_robots_txt returns the merged robots.txt after applying the AI policy; generate_schema returns JSON-LD scripts for each detected schema type; get_monitor_trend returns mention rate per brand, share of voice, breakdown by provider, competitor comparison, trends over time, and brief sample answers.
Data sent and privacy
Each tool makes HTTPS requests only to https://www.crawlreadiness.com , and sends nothing anywhere else. What leaves your machine includes: the website address you ask about (for generators, also the input content), the configured API key (if any), and the crawl-readiness-mcp/<version> user agent. The server stores nothing locally, runs no other programs, and does not read files; writing generated files into your project is done by the AI client with your permission. For the service's own data handling, see crawlreadiness.com/privacy.
The plugin's .mcp.json launches the published npm package and pins it to the same version as the repository (npx -y crawl-readiness-mcp@<version>), so the code running is this repository's code at that tag.
When to use it
Suitable for developers, SEO practitioners, and website operators who want to complete the "audit → fix → apply" loop inside an AI assistant, without copy-pasting between an editor and another SEO tool. Supports MCP clients such as Claude Desktop, Cursor, Cline, and Windsurf.
Limitations
Generator tools and monitoring tools require a free API key; the free tier includes 50 checks per month. Audit tools require no sign-up. The server itself stores no data and reads no files; file writing is performed by the AI client.
Other information
Docs: crawlreadiness.com/mcp; issue reports: github.com/Grubbomatic/crawl-readiness-mcp/issues; email: support@crawlreadiness.com. Crawl Readiness is a suite of AI-SEO tools that checks whether AI systems can access a website and helps fix blocked content. Every generator tool in this MCP server is also available on the web at crawlreadiness.com/tools. License: MIT © 2026 Crawl Readiness / Tundrastone. Release (maintainers): npm run release validates server.json and prints the release steps; the last step uses the mcp-publisher CLI, which is not part of this repository and must be downloaded from MCP Registry releases and placed on your PATH or in the repository root (it is already git-ignored in the repository root).
