os-experiment-log

Overview

The experiment log is the unified cross-cutting record for all agentic-os experiments. One file per run, all files in context/experiment-log/, with index.md as a queryable table of all runs.

context/experiment-log/
  index.md                                     ← one row per run (date, source, target, verdict)
  2026-04-25-verifier-os-architect-round1.md   ← from os-evolution-verifier
  2026-04-25-tester-os-architect.md            ← from os-architect-tester
  2026-04-25-os-improvement-loop-os-eval-runner.md    ← from os-improvement-loop
  2026-04-25-planner-0024.md                   ← from os-evolution-planner
  2026-04-25-survey-session.md                 ← from post_run_survey

Source Types and Result Kinds

Agents must check result_type in a log entry's header before parsing it:

`--source-type`	Produced by	`result_type`	Key fields
`verifier`	os-evolution-verifier	`qualitative`	PASS/PARTIAL/FAIL counts, HANDOFF_BLOCK validity
`tester`	os-architect-tester	`qualitative`	AC-1–4 pass/fail per scenario
`orchestrator`	os-improvement-loop	`numeric`	best_score, baseline, delta, KEEP/DISCARD counts
`planner`	os-evolution-planner	`qualitative`	workstream count, gaps identified
`survey`	post_run_survey	`mixed`	friction item count, north_star metric

Numeric entries (result_type: numeric) carry quantitative metrics suitable for trending and charting. Qualitative entries (result_type: qualitative) carry pass/fail verdicts and gap analysis prose. Mixed entries (result_type: mixed) carry both — agents must check which fields are present before parsing.

Phase 1 — Resolve Mode

Read the argument or invocation context to determine mode:

append --source-type TYPE: log a new run from a completed experiment
query <term>: search all files in context/experiment-log/ by keyword
summary: print aggregate stats across all runs, broken down by source type

Phase 2 — Execute

# After os-evolution-verifier run
python3 scripts/experiment_log.py append \
  --source-type verifier \
  --report temp/os-evolution-verifier/test-report.md \
  --session-id 2026-04-25-round1 \
  --target os-architect \
  --triggered-by os-evolution-verifier

# After os-architect-tester run
python3 scripts/experiment_log.py append \
  --source-type tester \
  --report temp/test_report_consolidated.md \
  --session-id 2026-04-25-tester \
  --target os-architect \
  --triggered-by os-architect-tester

# After os-improvement-loop run (numeric — has score delta)
python3 scripts/experiment_log.py append \
  --source-type orchestrator \
  --report temp/logs/run-log.md \
  --session-id 2026-04-25-os-eval-runner \
  --target os-eval-runner \
  --triggered-by os-improvement-loop

# After os-evolution-planner writes a task plan
python3 scripts/experiment_log.py append \
  --source-type planner \
  --report tasks/todo/0024-plan.md \
  --session-id 0024 \
  --target os-eval-runner \
  --triggered-by os-evolution-planner

# After a post-run survey
python3 scripts/experiment_log.py append \
  --source-type survey \
  --session-id 2026-04-25-session \
  --target session \
  --triggered-by human

# Query by term
python3 scripts/experiment_log.py query T2-D
python3 scripts/experiment_log.py query FAIL
python3 scripts/experiment_log.py query numeric

# Aggregate summary
python3 scripts/experiment_log.py summary

Phase 3 — Confirm and Report

After append:

tail -5 context/experiment-log/index.md

Report: "Logged to context/experiment-log/<filename>. Index updated."

After query: relay matching file names and their header blocks (date, source, target, verdict).

After summary: print the per-source-type breakdown verbatim.

Log Entry Format

Each file has a YAML-like header fence followed by the full report:

---
type: verifier
result_type: qualitative
date: 2026-04-25 15:12
session_id: 2026-04-25-round1
source: os-evolution-verifier
target: os-architect
verdict: 8P/0Pa/0F of 8
---

## Experiment — 2026-04-25 15:12 | verifier | os-architect

| Field | Value |
...

[full report content]

### Actions Taken
_[fill in: spec fix, new eval, new skill]_

Smoke Tests

Smoke 1 — Append verifier: Run python3 scripts/experiment_log.py append --source-type verifier. Confirm new .md file appears in context/experiment-log/ and index.md has a new row.

Smoke 2 — Query: Run python3 scripts/experiment_log.py query PASS. Confirm output lists at least one matching file with its header.

Smoke 3 — Summary by type: Run python3 scripts/experiment_log.py summary. Confirm output shows [verifier], [orchestrator] etc. sections with correct run counts.

Gotchas

Never parse result_type: mixed with numeric-only logic: The survey source type contains both friction prose and numeric north_star values. Always check result_type in the file header before assuming field presence.
temp/ is ephemeral: Call append immediately after a run completes, before any shell restart. The script exits with an error if the report file is missing rather than appending empty data.
Actions Taken is human-filled: The script writes a placeholder. An experiment log without response actions is an audit trail, not a learning record. Fill it in before the next run.
Duplicate index rows: If append is called twice for the same session, two rows appear in index.md. This is intentional (the file is append-only) but worth noting when querying.