SuperAPI AI Directory

agent-run-forensics

Agents & models

View repository

Updated 11-06 12:06

About this skill

Agent Run Forensics

Portability: 3 stdlib Python CLIs and one hook. Read-only, no network, no model calls. Secrets in transcripts are redacted in every output, with no switch to turn that off.

Agents are confidently wrong about their own history. Asked "why did you change config.yaml?", an agent answers from what it remembers, and memory drops the parts that matter: the test run that failed, the file a sed -i one-liner rewrote, the edit made after the last green run. The harness recorded all of it. This skill makes the answer come from that recording.

The rule: when a question is about something that already happened, read the trace before answering.

Workflow

# 1 - Find the run (skip if the question is about this session: the hook gives you its path)
python scripts/trace_finder.py --since 7d --touched config.yaml
# 2 - Read it: what happened, in order, and which files changed
python scripts/trace_timeline.py --trace <path> --files        # SILENT = changed, never named by the agent
python scripts/trace_timeline.py --trace <path> --grep timeout # or --errors, --file PATH, --around N
# 3 - Walk back from the step in question
python scripts/trace_timeline.py --trace <path> --why 9
# 4 - Before repeating any summary as fact, check it
python scripts/claim_checker.py --trace <path> --claim "passed:pytest" --claim "not-edited:config.yaml"

Step 3 prints two lists and you keep them apart in the answer:

Label Means Example
recorded The harness wrote it down when it happened the command, its exit code, the agent's words in the same message
inferred Derived now by a named rule prior-write, read-before-write, retry-after-failure, target-seen-in-output

Say "the trace shows" only for recorded edges. Name an inferred edge as inferred, with its rule. The agent's own stated reason is recorded as a claim, not as the cause.

Step 4 turns each sentence of a summary into a typed claim (ran:, not-run:, passed:, failed:, edited:, not-edited:, read:). STALE means the tests passed but files changed afterwards; PARTIAL means exit 0 came from the last command of a pipe or ; chain. Exit 4 is a contradiction: correct the account first.

Hard rules

  • No trace, no answer. If the finder returns nothing (exit 2), say the run is not recorded here. Do not reconstruct it.
  • Cite steps. Every factual sentence about the run points at a step number.
  • Recorded and inferred stay separate. Never merge them into one confident sentence.
  • Shell effects are inferred unless the harness diff (bashEditDiff) confirms them. A command's text says what it was meant to do, not what it did.
  • Exit 0 of a compound line is the last command's. pytest | tee log passing says nothing about pytest.
  • Quote no secrets. Outputs are redacted; do not paste raw transcript lines around the redaction.

Scripts

Script Role Exit codes
scripts/trace_finder.py Lists recorded sessions by repo, window, touched file, text 0 found · 2 none · 3 bad input
scripts/trace_timeline.py Numbered steps, --files table, --why STEP chain 0 · 2 filter matched nothing · 3
scripts/claim_checker.py Typed claims vs the recording 0 all supported · 2 weak · 3 · 4 contradicted

All support --help, --sample and --output json. trace_core.py is the shared parser. The plugin's UserPromptSubmit hook (hooks/forensics_hint.py) adds one line naming this session's transcript when a prompt asks about past work; disable it with AGENT_RUN_FORENSICS_HOOK=0.

References and assets