rag-mcp

Purpose

rag-mcp is a minimal, honest "corpus retrieval" MCP server, whose goal is to provide callers with traceable document retrieval capability: after vectorizing a document corpus, it returns passages with citations for a query, so that answers can be verified against sources. The project is designed to be embeddable in the manifest model of mcp-factory.

Main capabilities

• A single tool search_knowledge(query, k, doc_class=None): embeds the query, performs vector retrieval in the local corpus, and returns passages with citations (source + heading + chunk_index).
• Fully local, zero-cost embeddings: does not depend on a paid embedding API.
• Uses ChromaDB as the vector store.
• Incremental ingest: incremental by default. A manifest inside the storage directory records the SHA-256 of each file's decoded text; each run re-embeds only what actually changed, and cleans up residue that upsert alone cannot handle (chunks of deleted/renamed notes, trailing leftover chunks of notes that became shorter).
• Safety checks:
  • Citations: every hit carries source + heading + chunk_index.
  • Auth-scoped: results are restricted to the configured corpus root directory; sources that escape that root directory (absolute paths,.. traversal) are rejected.
  • Fail-soft: when the store is unavailable or empty, returns a structured error, and never throws an exception that would crash the caller's agent.
  • Bounded:k is restricted to [1, 20]; empty queries are rejected.
• Dependency versions are pinned (requirements.txt).

Protocol versions

Pinned to mcp==2.0.0, this is the first SDK version to implement MCP protocol revision 2026-07-28. The server simultaneously serves both protocol eras on the same stdio connection, and the client's first frame determines the choice: the 2026-07-28 revision cannot be reached through the initialize handshake; it belongs to the "modern" revision and is reached through server/discover or an inline _meta version marker. Era selection is automatic per connection, with no server-side switch.tests/test_protocol_version.py asserts both paths end to end, so a dependency rollback that causes the server to silently fall back to the old revision will fail in CI rather than quietly pass.

Keeping the index fresh (incremental ingest)

Incremental ingest is enabled by default. A manifest inside the storage directory records the SHA-256 of each file's decoded text; a run re-embeds only what truly changed, and cleans up what upsert alone cannot clean up (chunks of deleted/renamed notes, and trailing leftover chunks of notes that became shorter).

Measured on a live 2808-file / 26.6 MiB corpus (bge, CPU): no specific values are given in the material.

This is exactly what makes high-frequency scheduling affordable:reingest.bat is designed to run every 15 minutes, rather than once a day at 03:00—the latter would leave a note written at 03:05 invisible to search_knowledge for nearly 24 hours.

The manifest is trusted only when the run identity matches—embedder, embedding dimension, collection, and chunking parameters. Changing any one of these causes every file to be re-embedded, so an embedder replacement is never quietly applied halfway. A missing, corrupted, or mismatched manifest, or a manifest corresponding to empty storage, degrades to a full rebuild; in no case does it degrade to an incorrect skip.

Snapshot deduplication

The manifest's skip is a whole-file hash, so it cannot identify the duplication that truly harms retrieval: daily snapshot sequences (such as fleet-health-2026-07-23.md , etc.) repeat yesterday's passages verbatim within the file, while the file hash still changes. Measured on a live vault, a ## RED Bots status In fifteen consecutive files, five distinct values were taken, and byte-identical copies crowded the top-10, burying the document that explains it at position 16.

Therefore, ingestion still deduplicates at the chunk level, but only within date-bearing sequences, and only compared with the immediately preceding snapshot. The first occurrence is always embedded and retains its own date as its source; subsequent verbatim repetitions are not embedded but instead extend the survivor's repeat_dates metadata, which search_knowledge returns as snapshot_date / also_unchanged_on / snapshots_covered. Thus, "what it said on a given date" remains answerable—this is precisely why sequences are deduplicated rather than excluded. A value that changes and then reverts is retained, because it is a new fact rather than a repetition.

The scope is narrow, and the rule is written in rag_mcp/snapshots.py: the filename ends with -YYYY-MM-DD , there are at least 3 such files in the same directory and stem, and they are byte-identical under the same title. On the live corpus, this is 316 out of 2814 files, merging 842 out of 50428 chunks (17.5% of sequence chunks, 1.67% of the entire corpus), and it did not touch any ordinary notes. It can be disabled using --no-snapshot-dedupe .

--full rebuilds in place (ignoring the manifest, preserving storage);--clean deletes the storage first. Both still write the manifest, so the next run is cheap.reingest-clean.bat(weekly) is retained as a double-insurance reset.

Use as an MCP server

Register via mcp.yaml (validated against mcp-factory's Manifest loader). The tool is search_knowledge(query, k, doc_class=None); it reads the storage configured by the RAG_MCP_* environment variables.

Filter by document class

The metadata of each chunk carries a doc_class, set at ingestion time. It is "handoff" when the document's YAML frontmatter has a type: handoff or tags entry containing handoff (case-insensitive); or—since document frontmatter is optional and these session mirrors usually carry no frontmatter—when the file is directly under handoff_mirror_dir (default context/, configurable via ingest()'s handoff_mirror_dir= / handoff_mirror_basenames= parameters or the CLI's --handoff-mirror-dir / --handoff-mirror-basename flags) and named handoff.md / active.md / resume.md, or matches an anchored "handoff" filename token (e.g., handoff-2026-09-03.md, morning-dispatch-handoff.md)—never using bare substring matching, so documents whose title merely mentions the word (handoff-skill-redesign-spec.md) remain "note". Everything else defaults to "note". Passing doc_class limits the query to a single class, for example excluding the agent's own session/handoff bookkeeping from knowledge queries:

doc_class is a validated, case-sensitive enum—"note" or "handoff" (see rag_mcp.search.ALLOWED_DOC_CLASSES)—omitting it applies no filtering. Values outside this set (wrong case, typos, any other type) return a structured invalid_doc_class error, with the same shape as invalid_query. A doc_class that is syntactically valid but for which there are currently no matching entries in storage still soft-fails to an empty ok: true result.

An incremental ingestion run backfills doc_class (and any other metadata schema changes) onto already-embedded, unchanged chunks without re-embedding—see ingest.CURRENT_METADATA_VERSION. A store entirely older than the feature needs exactly one incremental run to give every chunk the correct doc_class, rather than a full --clean rebuild.

Test count: this is the total number collected and is enforced by CI against the live pytest summary (scripts/check_readme_counts.py, connected to .github/workflows/ci.yml). One test is platform-conditional—WINDOWS_ONLY 's tests/test_reindex_handle_release.py runs only on Windows—so Linux CI reports 263 passed, 1 skippedfor the same set of 264 tests; the gate compares passed plus skipped counts against the declared value, rather than comparing only passed counts, so this expected skip is not read as drift.

Commercial Support

Maintained by Jaimen Bell. For production-grade MCP integration, custom servers, or agent reliability work, see jaimenbell.dev.

Building your own MCP server? The MCP Starter Kit provides templates, a build handbook, and the packaging lessons learned from shipping this server.

Limitations and Not Yet Provided

• The material does not give specific pricing; for commercial support you need to contact jaimenbell.dev yourself.
• The material does not provide running examples beyond the install command, a list of supported file formats, performance on non-English corpora, or performance benchmark figures.
• The measurement data for incremental ingestion is only described as "measured on a live 2808-file / 26.6 MiB corpus (bge, CPU)"; no specific figures are given.