trace-grader

Skill: trace-grader

Grade an agent's trajectory, not just its output. A run can return the right answer via a bad path (force-pushed, retry-stormed, edited a file it never read) — "corrupt success" — and it can be flaky (passes once, fails on rerun). An answer-only checker is blind to both; TraceGrader inspects the path.

WHAT

Purpose and context

• Purpose: scores the agent's PATH (tool choice, ordering, retries, repeatability, grounding), not just the answer, to catch "corrupt success" and flakiness with a deterministic signal gateable in CI.
• Trigger (when to use):
• You have one or more agent runs (traces) and want to know how the answer was reached, not only whether it was reached.
• You need a repeatability gate: "does this task pass on ALL k reruns?" (pass^k).
• You want a deterministic, CI-gateable signal on tool choice / ordering / retries / step budget — before reaching for an LLM judge.
• Non-goals: does not grade the quality of the final answer; does not enable the agent-as-judge axis (STUB, not yet called) while the config is still verified:false; does not modify a flagged run.

Mental model

trace nguồn (traces.json · audit jsonl · session transcript) → parser → run = [{step, tool, args, observation, ok}] → rule checks (config) + pass^k → verdict per run (clean-pass · pass-with-warnings · corrupt-success · fail) → exit 3 nếu corrupt/flaky. Rule parameters live ONLY in the adapter harness/trace-grader.config.yaml (details in the Reference section below).

Input and output contract

	Field	Required?	Meaning
In	--traces / --audit / --transcript	yes (one of three)	canonical traces.json; audit jsonl (one run per session_id); Claude Code session transcript
In	--task	when using audit/transcript	assigns runs to a task to compute pass^k
In	--json	no	machine-readable output
In	harness/trace-grader.config.yaml	yes	forbidden tools, retry_threshold, step_budget, order_constraints, grounding.enabled
Out	verdict per run + flags + block pass^k	yes	report
Out	exit code	yes	3 if there is corrupt-success or the task is flaky

Rules and capabilities

• RULE-01 (MUST): The config file is the ONLY place rule parameters live. Same guess in two places = a leak.
• RULE-02 (MUST): Never present a guessed threshold as verified — the file stays verified:false until validated.
• RULE-03 (MUST): The agent-as-judge axis is OFF and uncalled until configured + verified; never fake an inferential score.
• RULE-04 (MUST): The audit parser cannot see per-step ok/observations, so retry_storm won't fire from a pure audit log — use full traces.json when you need failure-aware checks.
• Capabilities: read local trace/log/transcript files; read one adapter config file; do not call the model (judge off), do not write anything other than the report.

Failure boundaries

• Has run corrupt-success or pass^k = FAIL → exit 3 (failed for CI gate) — that is a signal, not a tool error.
• Source is only the audit log → cannot see ok/step-by-step observation, retry_storm does not fire → result partial, use traces.json in full if needed.
• Transcript has no task-verdict → run.ok is forced True → trust FLAGS, do not trust verdict delivered.
• Thresholds not yet checked on real traces → keep verified:false, do not present as verified.

HOW

Main workflow

Step	Type	Inputs	Action	Outputs/exit	Failure/next
W01	deterministic	runs	Collect traces (traces.json or --audit/--transcript)	trace source	audit only → partial (B01)
W02	judgment	project	Set the boundary: only edit harness/trace-grader.config.yaml	config	—
W03	deterministic	trace + config	Grade: run trace-grader.py	verdict + flags + pass^k, rc	rc 3 → W04
W04	judgment	report	Act on signal: investigate the flagged step / treat a flaky task as untrustworthy	conclusion	—
W05	judgment	real traces	(Later) verify adapter then flip verified:true	config verified	not enough evidence → keep false (B02)

Step-by-step details (source of truth for W01–W05):

1. Collect traces. Either emit traces.json in the schema above, or point --audit at a hooklib.audit() jsonl (one run per session_id).
2. Set the boundary. Edit harness/trace-grader.config.yaml ONLY — forbidden tools, retry_threshold, step_budget, order_constraints. Keep it the single place these values live.
3. Grade. Run the grader; read per-run verdicts + flags + the pass^k block.
4. Act on signal. corrupt-success = right answer, bad/unsafe path → investigate the flagged step. pass^k = FAIL = flaky → not safe to rely on.
5. (Later) verify the adapter. Validate thresholds against real traces, then flip verified:true. Only then consider enabling the agent-as-judge axis.

How to run

# bundled 3-case self-test (clean / corrupt / flaky)
python3 harness/scripts/trace-grader.py --self-test

# grade canonical traces, an audit log, or a full session transcript; --json for machine output
python3 harness/scripts/trace-grader.py --traces traces.json
python3 harness/scripts/trace-grader.py --audit .claude/audit/<date>.jsonl --task my-task
python3 harness/scripts/trace-grader.py --transcript ~/.claude/projects/<proj>/<session>.jsonl --task my-task

Exit code is 3 if any run is corrupt-success or any task is flaky (so CI can gate).

Branches

ID	Kind	Guard	Behavior	Skip / failure	Rejoin
B01	conditional_required	the source is a pure audit jsonl	can be graded but retry_storm does not fire; report the limitation	needs failure-aware → switch to traces.json or --transcript	W03
B02	capability_optional	judge model is configured AND config verified:true	only then consider enabling the agent-as-judge axis (judge_trajectory())	not sufficient yet → STUB, do not call	W04

Validation and stopping

Deterministic: python3 harness/scripts/trace-grader.py --self-test (3 cases clean / corrupt / flaky), verdict and exit code are decided by the engine. Needs review: the flip of verified:true after comparing the threshold with the real trace. Stop after W04; W05 is a later step, not run in the same round without a real trace.

Examples

• Positive: python3 harness/scripts/trace-grader.py --transcript ~/.claude/projects/<proj>/<session>.jsonl --task my-task on a session with an Edit to a file never Read → flag edited_without_read (medium), verdict pass-with-warnings, point out the step that was flagged.
• Boundary/failure: 3 runs of the same task, 1 failure → pass^k = FAIL → exit 3, the task is considered flaky; the run returns the correct answer but has forbidden_tool (force-push, high) → corrupt-success, exit 3.

Reference — What it is (build-now vs adapter)

Built now — deterministic, no unknowns:

• Trace schema (the contract): a run = list of steps {step, tool, args, observation, ok}.
• Parsers: canonical traces.json; the audit jsonl shape written by llmwiki/.claude/hooks/hooklib.audit() (and code-logger events.jsonl); OR a full Claude Code session transcript (--transcript) — the richest source: it recovers retrieval args (Grep pattern, Read file_path), observation, and per-step ok, so grounding + outcome are observable. Sidechain (sub-agent) lines are skipped; a transcript carries no task-verdict so its run.ok is forced True — trust the FLAGS.
• pass^k: k runs of one task → require ALL k to deliver; report pass^k.
• Rule checks (config-driven): forbidden_tool (high), out_of_order (medium, via must_precede), retry_storm (medium, >N consecutive same-tool failures), excessive_steps (low, > budget), edited_without_read (medium — grounding proxy: an Edit/MultiEdit to a file never Read/Grep'd/Write-authored first; gated by grounding.enabled).
• Verdict per run: clean-pass / pass-with-warnings / corrupt-success (delivered and a high-severity flag) / fail (not delivered).

Quarantined behind ONE adapter — harness/trace-grader.config.yaml, verified:false:

• the rule PARAMETERS (forbidden list, retry threshold, step budget, order constraints) — project-specific best-guesses, each # ASSUMPTION;
• the inferential agent-as-judge axis (task-completion / tool-rationale / planning rubric) — judge_trajectory() is a STUB, not called until a judge model is configured and the file is flipped verified:true.

Adapting later = edit that one file, never the engine.

Reference — Files

File	Role
harness/scripts/trace-grader.py	deterministic engine: schema, parsers, pass^k, rule checks, report, judge STUB
harness/trace-grader.config.yaml	the ONE adapter (verified:false) — every value a # ASSUMPTION
harness/tests/trace-grader.fixtures.json	self-test fixtures: clean / corrupt-success / flaky