web-analyzer-mcp

Purpose

web-analyzer-mcp is an MCP (Model Context Protocol) server for intelligent web page content analysis and summarization. It can serve as a backend service called by AI tools, helping extract clean, structured content from web pages and enabling question answering over web page content. The server is built using the FastMCP framework and is mainly aimed at developers who need to use web page content for analysis, question answering, or retrieval-augmented generation (RAG) scenarios.

Main capabilities

• url_to_markdown: Extract and summarize the key content of a web page. It uses a custom algorithm to analyze content importance, removes ads, navigation, and irrelevant content, retains only essential information (tables, images, key text), and outputs structured Markdown that is easy to analyze.
• web_content_qna: AI question answering based on web page content. It extracts relevant sections from the web page and, through intelligent chunking and relevance matching, uses OpenAI GPT models to answer questions.
• Intelligent content ranking: Algorithm-based content importance scoring.
• Retain only essential content: Remove clutter and keep key content.
• Multi-IDE support: Can be used with Claude Desktop, Cursor, VS Code, and PyCharm.
• Flexible model selection: Choose among GPT-3.5, GPT-4, GPT-4 Turbo, or GPT-5.

Usage

Prerequisites:

• uv (Python package manager)
• Chrome/Chromium browser (for Selenium)
• OpenAI API key (for question answering functionality)

Install via Smithery (for Claude Desktop): The material provides instructions for automatically installing web-analyzer-mcp via Smithery.

IDE/editor integration:

• Add to Claude Desktop's _config.json file; see the Claude Desktop MCP documentation for more details.
• Add the server using the Claude Code CLI.
• Add to Cursor settings (File > Preferences > Settings > Extensions > MCP).
• In JetBrains IDEs, go to Settings → Tools → AI Assistant → Model Context Protocol (MCP), click Command in the upper-left corner of the dialog, select the As JSON option from the list, add the configuration, and click OK. See the JetBrains AI Assistant documentation for more details.

Tool parameters:

• url_to_markdown: parameter url(string), the URL of the web page to analyze; returns clean Markdown content that preserves structured data.
• web_content_qna: parameter question(string), questions about page content; returns AI-generated answers based on page content.

Architecture

Content Extraction Pipeline:

1. URL validation—ensures the URL format is correct
2. HTML fetching—uses Selenium to handle dynamic content
3. Content parsing—uses BeautifulSoup to process HTML
4. Element scoring—a custom algorithm ranks content importance
5. Content filtering—removes duplicate and low-value content
6. Markdown conversion—generates structured output

Question Answering Pipeline:

1. Content chunking—intelligent text segmentation
2. Relevance scoring—matches content with the question
3. Context selection—selects the most relevant chunks
4. Answer generation—integrates OpenAI GPT

Use Cases

• Analysis tasks that need to convert web page content into clean Markdown.
• Scenarios that require AI question answering based on web page content.
• Integrate web page content analysis capabilities in IDEs such as Claude Desktop, Cursor, VS Code, and PyCharm.

Limitations

• For JavaScript-heavy websites, Chrome/Chromium support is required.
• The question answering feature requires an OpenAI API key.
• Rate limits exist to prevent abuse.
• Some websites may block automated access.

Other Information

• Future plans include: support for more content types (PDF, video), multilingual content extraction, custom extraction rules, caching of commonly used content, and Webhook real-time update support.
• This project uses the MIT license.
• This project is built on the FastMCP framework and is inspired by HTMLRAG technology.