SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformswet-mcp

wet-mcp

Web search via embedded SearXNG, content extraction and library docs indexing with hybrid FTS5 and semantic search using built-in Qwen3 embeddings.

Open website

About this tool

wet-mcp (the repository and CLI are now named wet, while the PyPI package name remains wet-mcp) is an open-source MCP server for AI agents, primarily offering three major capabilities: web search, content extraction, and library documentation. The current version is v3.x.

Purpose To provide AI agents with online retrieval, web content scraping, and local/online documentation indexing capabilities, enabling them to obtain real-time information, read web page body text, and retrieve library documentation.

Main capabilities

  • Web search: built-in SearXNG metasearch (Google, Bing, DuckDuckGo, Brave), supporting query expansion, TTL caching (1 hour for general / 5 minutes for time-sensitive content), standardized citation formatting, and a 200-token summary limit. Optional cloud search backends (Tavily, Brave, Exa, Kagi, OpenRouter, Firecrawl) can be configured via SEARCH_BACKENDS as a fallback chain.
  • Academic research: supports searching Google Scholar, Semantic Scholar, arXiv, PubMed, CrossRef, and BASE.
  • Library documentation: automatically discovers and indexes documentation, using FTS5 hybrid search, HyDE-enhanced retrieval, and version-specific documentation.
  • Content extraction: implements a 5-level strategy escalation chain via the n24q02m-web-core ScrapingAgent (basic_http → tls_spoof → rendering backends from BROWSER_BACKENDS (native / browserless / cf-browser-rendering) → optional key-required CAPTCHA), provides a markitdown bridge for low-level HTML/MD fallback, intelligent chunked structured output (clean text + markdown + JSON-LD + code blocks + metadata), batch processing (up to 50 URLs), deep crawling, and sitemaps.
  • Local file conversion: converts PDF, DOCX, XLSX, CSV, HTML, EPUB, and PPTX to Markdown.
  • Media: lists and downloads image / video / audio files. The analyze action was removed in v2.0.0; for visual/audio reasoning, use imagine-mcp.understand.
  • Anti-scraping: stealthstrategies can bypass Cloudflare, Medium, LinkedIn, and Twitter.
  • Zero configuration: built-in local reference embedding and reranking via fastretrieval, with no API key required. Optional cloud models can be configured per task via [models.embed|rerank|chat|jev_score] in ~/.wet/config.toml (default OpenRouter; a single OPENROUTER_API_KEY can serve all units).
  • Sync: sync indexed documents across machines via Google Drive (OAuth Device Code, no browser redirect required).

Usage Installation: pip install wet-mcp or uvx wet-mcp. The package installs two console scripts: wet (main command) and wet-mcp (legacy alias). A bare invocation (or any leading hyphen flag) starts the server; leading positional arguments are dispatched as subcommands. Transport modes: stdio (default, local) and HTTP (self-hosted, OAuth-gated). The HTTP endpoint uses Streamable HTTP, and on first connection the client prompts for authentication in the browser. Self-hosted HTTP server: development mode (uv, no-auth loopback) or persistent mode (docker). The MCP endpoint is http://127.0.0.1:8000/mcp，配置根目录为 ~/.wet/ (config.toml, docs.db, subs/). Do not start the server as a client subprocess; instead, register the endpoint on the client. Configuration: [models.embed|rerank|chat|jev_score] in ~/.wet/config.toml are configured per task (each with its own base_url + api_key + model, OpenAI specification). Examples provide an OpenRouter example; any OpenAI-compatible endpoint works, including local Ollama. Keys may be left empty and injected at startup via HULL_EMBED_API_KEY / HULL_RERANK_API_KEY / HULL_CHAT_API_KEY / HULL_JEV_SCORE_API_KEY. Search backends: SEARCH_BACKENDS is an ordered runtime fallback chain, defaulting to searxng (local or external via SEARXNG_URL), and can be configured with key-requiring tavily / brave / exa / kagi / openrouter, optional-key firecrawl, and credential-free duckduckgo / startpage. Browser rendering backends: BROWSER_BACKENDS (CSV, escalation chain) selects the headless rendering stage for extract: native (in-process chromium, zero-config default), browserless (self-hosted rendering service, requires BROWSERLESS_URL + BROWSERLESS_TOKEN), cf-browser-rendering (Cloudflare Browser Rendering, requires CF_ACCOUNT_ID + CF_BROWSER_RENDERING_TOKEN). An empty chain falls back to native. Setting CAPSOLVER_API_KEY appends an optional, key-requiring CAPTCHA tier as the final escalation step. Robots policy: Setting RESPECT_ROBOTS_TXT=true enforces robots.txt in the extract policy chain and in Crawl4AI-supported crawl, sitemap, and list_media actions. Default false. Stealth tier and seed identity (optional): Appending invisible to BROWSER_BACKENDS adds a stealth Firefox engine tier as the final escalation step (requires pip install "wet-mcp[invisible]"). Seed identity via WET_IDENTITY_SEED, and install the wet[identity] extra. STEALTHFOX_BINARY points to a patched Firefox build to skip the engine download; IDENTITY_PROFILE_DIR retains a persistent browser profile per namespace. Disabling local fallbacks: You can opt out of heavyweight in-process local fallbacks per capability: DISABLE_LOCAL_BROWSER, DISABLE_LOCAL_SEARCH, DISABLE_LOCAL_EMBED, DISABLE_LOCAL_RERANK. Document sync: SYNC_ENABLED (default true), GOOGLE_DRIVE_CLIENT_ID (required for sync), SYNC_FOLDER (default wet), SYNC_INTERVAL (default 300 seconds). DOCS_DB_BACKEND=cf-d1 disables GDrive/S3 file sync; SYNC_S3_BUCKET selects S3; SYNC_ENABLED=false disables both. HTTP self-hosting: MCP_TRANSPORT=http, PUBLIC_URL=<your-domain>. The settings form is gated by MCP_RELAY_PASSWORD; multi-user deployments require CREDENTIAL_SECRET, MCP_JWT_SIGNING_SECRET, MCP_DCR_SERVER_SECRET. Deploying to Cloudflare: Requires a Cloudflare account on the Workers Paid plan (required for Containers, D1, Vectorize) and the wrangler CLI. Steps include cloning the repository, creating a D1 database and applying migrations, creating a Vectorize index and KV namespace, building a slim HTTP image and pushing it to the Cloudflare managed registry, setting secrets, wrangler deploy, and completing in the browserSetup complete. Storage mapping is MCP_STORAGE_BACKEND=cf-kv, DOCS_DB_BACKEND=cf-d1, and Vectorize.

Use cases

  • AI agents need web search and web content extraction.
  • Need to retrieve repository documentation, academic papers, or versioned documents.
  • Need to convert local documents to Markdown for use by agents.
  • Need to sync indexed documents across machines.
  • Need to run with zero configuration in an environment without an API key.

Limitations

  • media(action="analyze") has been removed in v2.0.0; visual/audio/video analysis requires the understand action of imagine-mcp.
  • Public OCI image publishing has stopped; new container deployments must be built from source or use the Cloudflare hosted registry.
  • No-auth mode rejects non-loopback binding; the server is limited to localhost until switched to token authentication.
  • When the chat unit key is not configured, LLM features (such as extract(action="agent")) are unavailable.
  • Deploying to Cloudflare requires the Workers Paid plan; the free tier does not include Containers, D1, and Vectorize.
  • Specific billing, pricing, and performance data are not yet provided.

Related tools

Browse category