SearchForge

Purpose

SearchForge is a free and open-source web search API and MCP server for LLMs, AI agents, and retrieval-augmented generation (RAG) pipelines. It is designed to provide a predictable retrieval layer so that projects do not have to be forced into integrating paid search vendors. It does not generate answers, does not hide citations, does not scrape public SearXNG instances, and does not send telemetry data.

Key Capabilities

• Capability-routed search: The default auto category detects code, academic, and current events/community signals, and queries matching source families as well as a web fallback. Explicit categories avoid querying irrelevant providers. Explicitly providing a providers list overrides category routing and is suitable for evaluation scenarios.
• URL to Markdown:read_url accepts only public HTTP(S) URLs, and rejects credentials, localhost, private IP literals, and non-web protocols. Responses have size limits, timeouts, and caching.
• Diagnose the entire retrieval path:doctor performs real, bounded probes and reports each provider's access tier, capabilities, latency, and errors. When a source fails, it produces a degraded status rather than a misleading all-or-nothing status.
• Covered sources: GitHub, Crossref, Hacker News, Wikipedia, and private SearXNG. GitHub tokens are used only to increase public API quota; Brave is an optional backend that requires a key. Broad, independent web metasearch is provided by the bundled SearXNG stack.
• Free and self-hosted first: All zero-key adapters are enabled by default.

How to Use

Try it without cloning: The first run downloads and builds the package from GitHub. Unless a category is selected, search uses intent-aware auto routing by default.

Zero-key local CLI: Full web search is possible with private SearXNG. This method starts SearchForge on port 3000 and a JSON-capable private SearXNG on port 8080. Before exposing this stack externally, you need to change the SearXNG secret, set SEARCHFORGE_API_KEY, and terminate TLS at a trusted proxy.

MCP server: It can be run directly from GitHub without cloning. SearchForge has been published in the official MCP Registry as io.github.divyanshu-iitian/searchforge. For MCP clients that support Docker commands, you can directly run the registry-backed OCI image.

MCP tools: SearchForge exposes three stdio tools:

• web_search — routed, directly citable structured search
• read_url — fetch clean Markdown from a public URL
• search_status — real-time capability and latency report

The search and status tools return both MCP structured content and readable text.

REST interface: An OpenAPI contract is provided. For other endpoints, refer to the full OpenAPI contract. An alternative method is needed before the npm release.

Suitable Scenarios

• Provide controllable web retrieval capabilities for LLMs or AI agents
• A retrieval layer in RAG pipelines
• Projects that need to avoid vendor lock-in and prefer free or self-hosted paths
• Search evaluations that need intent-based routing for code, academic, current events/community, and similar categories
• Convert public web page content into LLM-usable Markdown

Limitations

• SearchForge intentionally does not configure public SearXNG instances because they often disable JSON or restrict automated traffic; the Docker stack is a stable free path.
• The page content of search results is untrusted input and must be separated and protected with prompt injection defenses.
• The built-in cache and rate limiting are process-local; multi-replica deployments should use shared infrastructure.
• Provider response bodies and credentials do not appear in error reports.
• healthz only proves that the process is alive;/v1/doctor Used to check dependencies.
• The production environment must set an API key before binding to a public interface.
• Each idempotent provider call has an independent abortable timeout; a single failure will not erase healthy results. Tracking parameters are removed before deduplication, and each contributing provider is retained in sources .

Design Principles

• Evidence takes precedence over generated answers
• Free and self-hosted paths take precedence over vendor lock-in
• Partial results are better than complete failure
• Honestly report capabilities and quotas
• Stable contracts and clear provenance
• Telemetry is not sent by default

The project uses the MIT license, with copyright owned by Divyanshu.