ciagent

CIAgent is an open-source testing tool for AI agent evals whose core claim is: stable eval scores do not mean a stable system. The tool helps users determine whether their agent's evals are "lying."

Purpose

CIAgent is used to uncover hidden instability in eval results during agent development and CI pipelines. A single eval may produce seemingly solid overall scores like 96% / 95% / 96%, yet the decisions for individual queries underneath may be flipping every time, with errors simply canceling each other out in different places. CIAgent exposes these flips by running multiple times, and attributes each flip to the responsible layer, turning the problem into a dispatchable work item rather than a disquieting number.

Main capabilities

1. Stability reports and flip attribution: Use --runs N to run the same eval multiple times and observe which decisions flip. Attribution is structural rather than guesswork: deterministic checks cannot flip when the output is the same, while the sub-decisions of each layer are compared simultaneously.

   • agent-variance: the agent produced different outputs, and the prompt, retrieval, or temperature should be fixed.
   • judge-flake: the output (or the results of all deterministic checks) is the same, but the LLM judge changed its mind, and the grading rubric should be fixed or deterministic checks should be used instead.
   • infra-error: the judge API call failed, which is a retry issue. The console shows the observed facts; pass@k/pass^k estimates appear in the JSON output and are labeled as estimates. The exit code is 0 for flaky-but-passing, making it easy to adopt without breaking CI; add --fail-on-flaky when ready to gate on it.

2. LLM judge auditing (judge-audit): if an LLM judge shares context with the agent, it inherits the agent's blind spots—when retrieval is empty, the agent answers from thin air, and the judge reads the same blank and passes it. judge-audit re-scores an already recorded baseline using existing ground truth (without re-running the agent), providing: a disagreement matrix between the judge and deterministic checks (the key row being answers the judge passes but hard fact checks do not), retest stability (judging the same answer --repeats times; flips on identical input are the judge's own noise), and agreement and Cohen's κ under human labels (--labels). The verdicts are TRUSTWORTHY / NEEDS CALIBRATION / UNRELIABLE, and the argument is one-directional: a judge that fails where verifiable should not be trusted where unverifiable either.

3. Deterministic checks first: most critical failures involve hard facts (product names, prices, version numbers), which can be verified deterministically at zero cost. CIAgent runs deterministic checks first and uses the judge as a last resort:

   • Code fact checks: expected_in_answer, not_in_answer, regex_match, json_schema, zero LLM calls, zero jitter, identical verdict every time.
   • Path checks: whether the agent called the tools it should have; missing expected tools raise a warning, and forbidden tools appearing cause failure.
   • Cost budget: number of LLM calls, token count, and dollar cost per query.
   • LLM judge (llm_judge rubric, optional): used only for answers that genuinely require judgment, and evaluated after all deterministic checks.

4. Mine checks from the knowledge base: there is no need to hand-write fact checks; hard facts can be extracted from the knowledge base (prices, rates, SKUs, "30 days", etc.) as variant-set assertions, and each candidate is first validated against the recorded reference answer—a check that would make a known-good answer fail is rejected before being presented. One LLM call at authoring time, after which the check runs forever for free.

5. Layered evaluation: ciagent test evaluates each query through 4 layers—correctness, path, retrieval, cost. The retrieval layer reads the captured results of retrieval tools and issues deterministic warnings for empty retrieval, missing sources, and count lower bounds; when results are not captured or cannot be parsed, it performs a SKIP (never guesses).

6. Integration and adaptation: natively adapts to OpenAI, Anthropic, and LangGraph; can import trace data exported by any stack—OpenTelemetry (openllmetry, Google ADK), Langfuse, LangSmith, and has been validated against real captures from OpenAI, Anthropic, CrewAI, Google ADK, and Claude Agent SDK. Can run inside pytest.

7. Onboarding for coding agents: CIAgent is provided as a Claude Code plugin with two skills—onboard (write adapters, record the reference baseline, generate specs, validate) and check (run the correct tests after every agent change and triage failures by their source of flip).The adapter needs only one function: (query: str) -> str, and trace capture is done automatically.

8. More features (topics listed in the documentation): simulation (multi-turn conversation scenarios, scripted for CI, generative personas as discoverers), cost tracking (budgets and spike detection), golden traces (recording baselines, diff regression), importing production traces (OTel/Langfuse/LangSmith/ATIF), golden promotion, simulated worlds (freezing the tool traffic of failed runs and replaying it deterministically), MCP servers, world file format (with JSON Schema), failure maps (runnable, mapping OWASP's agent failure modes), CI/CD integration (GitHub Actions), LangGraph support, metrics reference.

Usage

There is corresponding documentation for installation and first run (quickstart), simulation, stability testing, judge auditing, generation checks, writing tests, cost tracking, and more. The demo requires no installation, API keys, or configuration; a single command runs the built-in demo suite three times on synthetic traces. Zero API key trial: CIAGENT_MOCK_FLAKY=1 ciagent test --mock --runs 3 . Mock mode (ciagent test --mock) runs the entire evaluation suite with zero API keys and zero cost, and can be used to gate every PR. When there are no golden queries yet, you can use ciagent init --generate to scan the code and generate a starting spec; when set up by a coding agent, you can have it "set up CIAgent for this repository", and it will use ciagent bootstrap --yes to record a golden baseline and ciagent test --runs 3 to verify.

Basic workflow: write golden queries (what the agent should handle and what it should refuse), then use ciagent test to evaluate through four layers: correctness, path, retrieval, and cost.

Applicable scenarios and limitations

Suitable for agent teams that use LLM judges as their primary evaluation method and are worried that evaluation results are unreliable, as well as engineering scenarios that want low-cost gating in CI and replace part of LLM judging with deterministic checks. Runnable examples (DemoAgents) are provided for OpenAI, Anthropic, and LangGraph agents.

Limitations and notes: the tool comes from a real study—LLM judgesAbout one in seven passing answers is wrong, while deterministic checks caught 8/8; see STUDY.md for the methodology and raw data. The demo simulates a flaky agent; pointing it at your own agent requires configuration according to the spec. pass@k/pass^k are estimates. Other details (such as specific prices, platform availability, performance metrics, etc.) are not provided in the materials.

License

Apache 2.0.