Diffcontext

Purpose

DiffContext is a context compiler for LLM coding agents. When you ask an assistant to modify a function in a 50,000-line project, you usually face three bad choices: paste the entire repository (it does not fit, and the model performs worse in an oversized context), paste only that one function (the model will break the three callers it has never seen), or grep by name (grep cannot find the subclass that overrides it, nor the handler that receives it via functools.partial). DiffContext is the fourth choice: parse the repository once into a real dependency graph, then for any change select the few functions that truly matter and package them into the smallest usable prompt.

It is aimed at people connecting LLMs to real codebases - agent loops, PR review bots, CI checks - that is, scenarios where you must decide what to put in the prompt while the repository is too large to send as a whole.

Main capabilities

• Given a Python repository and a change (git diff, branch, or a single function name), return the set of functions needed to complete the change, including callers that will break, subclasses that override the function, and tests that cover it.
• Fit the selected code within a specified token budget and tell the model what was omitted.
• Self-evaluation: through diffcontext verify --from-history 20 --calibrate mine test cases from the repository git history, score retrieval, and print NULL RESULT instead of decorative numbers when the tool is not suitable for your repository.
• Retrieval quality is measured against real co-change pairs mined from git history and rerun as a CI gate on every push to prevent silent quality regression.
• Can be used as a library, providing an incremental API (idx.update([...])), structured output, and pluggable tokenizers; see docs/ARCHITECTURE.md for details.
• Provides MCP integration (Claude Code / Cursor / Windsurf), with configuration in docs/MCP.md; provides a FastAPI service and Web UI (diffcontext-service/); provides retrieval pipeline tracing (observability/).

Usage

No runtime dependencies, supports Python 3.9+.

MCP integration (Claude Code / Cursor / Windsurf): see docs/MCP.md for server configuration.

Install from source for development: (the installation command material is not fully provided here, so no specific command is provided for now).

For more commands see USAGE.md. For production recipes see docs/USE_CASES.md.

Self-evaluation: do not trust its benchmarks, run your own (2 minutes):

diffcontext verify --from-history 20 --calibrate

This command mines test cases from your repository git history, scores retrieval, and prints NULL RESULT instead of decorative numbers when the tool is not suitable for your repository.

Use as a library: incremental API (idx.update([...])), structured output, pluggable tokenizers; see docs/ARCHITECTURE.md.

Applicable scenarios

• Connecting LLMs to real codebases: agent loops, PR review bots, CI checks.
• Need to put only change-relevant code into the prompt while being constrained by a token budget.
• Code context selection and packaging for Python projects.

Limitations

• Not a code generator. It selects and packages context; the model writes code.
• Not precision-first. Under the default top-k, average precision is below 0.1. If you pay by token, you can use --cutoff gap.
• Multilingual support is not yet available. Python is fully supported. TypeScript/JS (ESM) is a working prototype; CommonJS is a measured failure mode.
• Cannot replace reading the code. Static analysis has blind spots; see below and docs/BENCHMARKS.md.
• Static analysis has limits: topic sibling functions with no call relationship, conceptual links across subsystems (all methods score 0/20), and dynamic dispatch are measured blind spots.
• When in doubt, before fully trusting "no callers found", first run grep -rn "function_name(" --include="*.py" ..

Effect description (measurement results in the materials)

On 128 ContextBench Python tasks, judged by each repository's own test suite (no LLM-as-judge), context roughly quadruples pass@1: 5.5% → 25.8%, exact McNemar p < 0.0001. Both qualifications appear in benchmarks/contextbench/RESULTS.md §6: (a) the seed function provided to each group is an oracle—extracted from the gold patch—so this measures "given correct localization, does context quality matter," not end-to-end problem solving (localization is provided free to all groups); (b) 121 of the 128 valid tasks are django, so this is largely a django result. The honest companion conclusion: the three context variants (default / gap / depboost) are statistically indistinguishable from one another, p = 0.36–0.81. The win is context versus no context, not this selector versus that selector.

Retrieval quality is measured on 701 real commits across 9 Python repositories and rerun as a CI gate on every push. Compared with grep under the same token budget, grep's recall plateaus at 0.215 after more than 4k tokens, while DiffContext reaches 0.576 at 8k (2.7x). The honest other side: average precision under the default top-k is below 0.1—most retrieved symbols are supporting context, not the exact co-change set.--cutoff gap Truncate at the largest score drop, trading about 30% recall for about 4x precision (co-change benchmark; on ContextBench, 2.2x / about 14%).

The authors audited their own benchmark, and three claims failed: calibration—the only citable number (r=0.274, n≈25) was measured on a contaminated index, and after a clean retest at n=1,080 the old score gave r=0.016 (p=0.60): no relationship at all; fixed by shrinking toward "don't know" → r=0.287 (p=0.0001), a ranking signal rather than a probability. Blend weights—the published [0.5, 0.35, 0.15] failed leave-one-repository-out testing; it is now [0.3, 0.5, 0.2]. Dense baseline—the TF-IDF substitute exaggerated dense retrieval (0.664, beating BM25 5/5); the real MiniLM encoder scores 0.597, beating BM25 only 2/5. The full record is in docs/auditing-my-own-benchmark.md, and the original report is in benchmarks/RIGOR_REPORT_2026-07.md.

Documentation

• docs/ARCHITECTURE.md — pipeline, module diagram, agent API
• docs/BENCHMARKS.md — all numbers, downstream pass@1, limitations
• docs/MCP.md — MCP server for Claude Code / Cursor / Windsurf
• docs/ROADMAP.md — prioritized plan with measurement motivation
• diffcontext-service/ — FastAPI service + Web UI
• observability/ — retrieval pipeline tracing
• CONTRIBUTING.md — setup, CI gates, adapter development