whatbroke is a command-line tool for comparing behavioral differences between two runs of an AI Agent, provided by the GitHub project arthi-arumugam-git/whatbroke. Its core idea is: plain text diff cannot find problems at the Agent behavior level—for example, the model may say "your subscription has been canceled" while actually silently skipping the cancel_subscription call. The text looks fine, but the behavior is already broken. whatbroke is designed precisely to solve this kind of problem.
Main capabilities After changing models, fine-tuning prompts, upgrading framework versions, or modifying tool descriptions, run whatbroke to see the specific changes: which tool calls disappeared, which parameters drifted, where cost and latency changed, and which outputs flipped. The tool aligns the two runs by id, aligns tool calls within each run, and then reports the differences. When destructive changes occur, the exit code is 1; use --fail-on changed to set a stricter gate, or use --fail-on never to only generate a report.
Multi-sampling and noise distinction An Agent will not do exactly the same thing twice, so a single before-and-after comparison may attribute the Agent's own inherent noise to the change. You can record the same scenario multiple times and add a suffix after the run id. whatbroke recognizes the suffix, compares each pre-change sample with each post-change sample, and gives a ratio for each finding. Items that also fluctuate between the two baseline samples are downgraded to flaky information, because the Agent behaved that way before this change; destructive findings whose occurrence rate is less than half of the pairings are softened to warnings, and only the rest are true signal.
Recording traces The trace format is deliberately designed to be very simple: JSONL, one event per line, which can be written in any language within ten minutes. The fastest way to get it is to use a proxy, with zero code changes, in any language. Point the Agent at the proxy and run as usual, and every LLM call, tool call, and final answer will fall into the trace; streaming works, and responses pass through as-is. If you are driving multiple scenarios, you can send an x-whatbroke-run header on each request to name the run. The Node SDK can wrap existing clients and record automatically, rec.wrapAnthropic(client) also works for the Anthropic SDK; in other cases you can use rec.llmCall(), rec.toolCall(), rec.output(), rec.end(), or write your own JSONL. The Python approach has the same shape (pip install whatbroke-recorder, zero dependencies), and rec.wrap_anthropic(client) likewise supports both synchronous and asynchronous clients. Traces recorded from Python, Node, and proxies can be compared with each other.
Importing existing traces If you are already tracing an Agent, you can use whatbroke import to convert existing content into comparable JSONL without re-recording. The format is auto-detected from the file, --format otel|langfuse|langsmith can force it, --run <name> sets the run name, and -o selects the output path. The import process prints what the source data does not carry (such as cost and tool parameters), so you know which findings cannot be shown.
Behavior contracts Diffing requires two traces, but sometimes you only need to declare conditions that must always hold and check them on every run. A contract is a JSON file committed alongside the code, rules apply to every run, and the runs mapping can be overridden by scenario. Don't want to write it by hand? You can learn from a good baseline and generate a starting contract containing the tools called by each sample, a cost/latency budget of the observed maximum plus 50%, and the expected state. The report prints the contract hash, making it easy to distinguish in CI logs whether "the Agent changed" or "someone changed the contract." Sampled traces report violation rates by rule rather than failing outright. The exit code is 1 on any violation (can be disabled with --fail-on never), and --md and --json output are likewise supported.
Watch mode Iterating on prompts means repeatedly running the same diff. whatbroke watch can pin the baseline, point the recorder at the current trace, and re-render the diff in the terminal whenever the file changes. The current file does not have to already exist; half-written lines are fine too, and the watcher will wait for writes to stabilize and retain the last good report. Ctrl-c stops it, and --no-clear preserves scrollback.
CI integration Both commands can run as composite actions, the Markdown report appears in the job summary, and before-and-after diffs are also supported. In CI, add --fail-on changed for diff, parameter drift will cause abuild failure along with breaking findings.
Relationship to evaluation tools An evaluation suite scores each version against grading criteria, whereas whatbroke answers a different question: what exactly changed between these two versions, down to the tool-call level, without writing grading criteria or paying for a judge model. It is suitable for running five minutes after a new model is released, and then deciding whether the evaluation suite still needs to run.
Features and limitations Deterministic, offline, no API key required, no account required, and traces never leave the local machine. Supports multi-sample runs, so unstable behavior appears as volatility rather than noise; proxy capture requires no code changes; provides OpenTelemetry, Langfuse, and LangSmith trace import; provides behavior contracts as a guardrail when there is no baseline; provides a Python recorder; provides watch mode; semantic output comparison is optional and requires bringing your own key. Known limitations: in some import sources, tool parameters and model outputs are optional attributes, and if they were not captured, the corresponding comparison is empty; the Langfuse free tier has no flat-file export, so you must export via the SDK first; Langfuse latency is recalculated from timestamps because the exported latency field's units vary across integration versions. More information on specific platform support and pricing is not currently available in the materials.
