Turn wiki goldens into a CI gate: cheap deterministic asserts decide most goldens, a committed baseline catches regressions, and the one expensive thing (an LLM judge) is quarantined behind a single adapter file.
WHAT
Purpose and context
- Purpose: make wiki golden pages a regression suite checked in pure code (no model), with a committed baseline and a CI exit code that fires only when a metric drops below it.
- Trigger (when to use):
- You want wiki pages to act as a regression suite an agent (or PR) cannot silently break.
- You have golden input→expected pairs and want them checked in pure code, no model needed.
- You need a CI step that exits non-zero only when a metric drops below a committed baseline.
- User says "wikieval", "eval suite for the wiki", "regression gate for goldens", "assertion cascade", "block CI on eval drop", or
/wikieval(model-invocation disabled — user invokes).
- Non-goals: never calls an LLM (tier 3 is reported
needs-judge, not judged); does not generate candidate outputs (they come from--outputs); does not author goldens' content for you.
Mental model
golden (input · expected · rubric? · asserts?) + candidate output → cascade: tier 1 asserts → [tier 2 similarity] → [tier 3 needs-judge] → per-golden pass+score → baseline (write) | regression diff (check). Cheap tiers short-circuit; the expensive one sits behind one adapter file.
Input and output contract
| Field | Required? | Meaning | |
|---|---|---|---|
| In | goldens llmwiki/wiki/sources/evals/*.md |
Yes | YAML frontmatter input, expected, optional rubric, optional asserts, optional id; also R9 type + R2 ## Origin |
| In | --outputs <outputs.json> |
Yes | candidate outputs per golden — makes the suite runnable without any model |
| In | mode | No | none = report; --write-baseline; --check |
| In | harness/wikieval.config.yaml |
No | adapter config (verified: false): judge model, rubric prompt, embedding.backend |
| Out | report | Yes | per-golden pass/fail/score/needs-judge |
| Out | harness/metrics/eval-baseline.json |
with --write-baseline |
committed per-golden pass + score |
| Out | exit code | with --check |
0 = no regression; 2 = a metric dropped below baseline (+ regression diff) |
Rules and capabilities
- RULE-01 (MUST): If a golden declares
asserts, tier 1 alone decides pass/fail. Unknown ops fail closed. - RULE-02 (MUST): tier 2 similarity is only reached when a golden has no asserts; skipped + noted unless
embedding.backendis set (difflib= stdlib lexical, offline; a real model id stays skipped until the adapter is wired). - RULE-03 (MUST): the
judge(output, rubric) -> {score, reason}stub is NEVER called in deterministic mode; escalated goldens are reportedneeds-judge. - RULE-04 (MUST):
--checkis a deterministic diff against the committed baseline; exit 2 only on a drop. - RULE-05 (MUST): the adapter boundary is one file — finalizing the judge changes
harness/wikieval.config.yaml+judge()only; the rest of WikiEval does not change. - Capabilities: read goldens + candidate outputs + adapter config; write baseline file; no network, no model.
Failure boundaries
- An assert fails → golden failed (report); with
--check, drop vs baseline → exit 2 + regression diff. - Unknown assert op → failed closed (not skipped).
- Golden without asserts and no
embedding.backend→ tier 2 skipped + noted; rubric escalation →needs-judge(not a pass, not a fail). - Missing/invalid
--outputsfile → blocked: cannot run without candidates.
HOW
Main workflow
| Step | Type | Inputs | Action | Outputs/exit | Failure/next |
|---|---|---|---|---|---|
| W01 | deterministic | goldens, --outputs |
Load goldens + candidate outputs | pairs per golden | missing outputs → blocked |
| W02 | deterministic | pair, asserts |
Tier 1 deterministic asserts | pass/fail if asserts declared | unknown op → fail closed |
| W03 | deterministic | pair, config | Tier 2 similarity (only if no asserts) | score or "skipped + noted" | backend unset → skip |
| W04 | deterministic | rubric | Tier 3 → report needs-judge (judge never called) |
needs-judge |
— |
| W05 | deterministic | results, mode | Report / --write-baseline / --check diff vs baseline |
report, baseline file, exit 0/2 | drop → exit 2 |
Step-by-step details (source of truth for W01–W05):
The assertion cascade (cheap → expensive, short-circuits)
- tier 1 — deterministic asserts (build-now core):
equals/contains/icontains/not-contains/regex/is-json/is-sql-ishrun on the candidate output. If a golden declaresasserts, tier 1 alone decides pass/fail. Unknown ops fail closed. - tier 2 — similarity (OPTIONAL): only reached when a golden has no asserts. Skipped + noted
unless
embedding.backendis set in the config (difflib= stdlib lexical, offline; a real model id stays skipped until the adapter is wired). - tier 3 — LLM-rubric judge = THE ADAPTER (
verified: false). The stubjudge(output, rubric) -> {score, reason}is NEVER called in deterministic mode; escalated goldens are reportedneeds-judge.
Goldens
Markdown under llmwiki/wiki/sources/evals/ with YAML frontmatter: input, expected,
optional rubric, optional asserts (quote each: 'contains:foo', 'regex:^...$',
'is-json', 'equals:bar', 'is-sql-ish'). Optional id (defaults to filename stem).
Each golden also obeys R9 (frontmatter type) and R2 (## Origin).
Run it (no model required)
# run the cascade with a fixed candidate set, print a report
python3 harness/scripts/wikieval.py --outputs harness/evals/wikieval-outputs.example.json
# write / refresh the committed baseline (per-golden pass + score)
python3 harness/scripts/wikieval.py --outputs <outputs.json> --write-baseline
# CI gate: re-run and exit 2 if any metric dropped below baseline (deterministic diff)
python3 harness/scripts/wikieval.py --outputs <outputs.json> --check
Baseline: harness/metrics/eval-baseline.json. Adapter config: harness/wikieval.config.yaml.
Branches
| ID | Kind | Guard | Behavior | Skip / failure | Rejoin |
|---|---|---|---|---|---|
| B01 | capability_optional | golden has no asserts AND embedding.backend set |
tier 2 similarity (difflib offline) |
backend unset or real model id not wired → skipped + noted | W04 |
| B02 | user_optional | --write-baseline |
write/refresh harness/metrics/eval-baseline.json |
— | end |
| B03 | user_optional | --check |
deterministic diff vs baseline, exit 2 on drop | no baseline → run --write-baseline first |
end |
Validation and stopping
Everything is code-checked: asserts, cascade, baseline diff and exit code are deterministic. Only tier 3 needs review (needs-judge) and stays out of the pass/fail decision until the adapter is finalized.
Self-test (build-now slice)
OUT=harness/evals/wikieval-outputs.example.json
python3 harness/scripts/wikieval.py --outputs $OUT --write-baseline # both goldens PASS
python3 harness/scripts/wikieval.py --outputs $OUT --check # exit 0 — no regression
# flip one output to break an assert, then:
python3 harness/scripts/wikieval.py --outputs <broken.json> --check # exit 2 + regression diff
Examples
- Positive:
python3 harness/scripts/wikieval.py --outputs harness/evals/wikieval-outputs.example.json --checkafter writing baseline → both goldens PASS → exit 0. - Boundary/failure: flip one candidate so
'contains:foo'no longer matches →--checkexits 2 with the regression diff naming that golden. - Boundary: golden with only
rubric, no asserts, noembedding.backend→ tier 2 skipped + noted, reportedneeds-judge; judge not called.
Reference — Adapter boundary (build-now-adapt-later)
- Built + tested now: tier-1 asserts, the cascade, baseline write +
--checkregression diff. - Quarantined (
verified: false):harness/wikieval.config.yaml(judge model + rubric prompt, embedding backend — each a flagged# ASSUMPTION) and thejudge()stub. - Finalize: wire
judge()to the configured model, run the self-test, flipverified: true. One file changes; the rest of WikiEval does not.
Reference — Files
harness/scripts/wikieval.py— the eval engine.harness/wikieval.config.yaml— the one adapter (verified: false).harness/metrics/eval-baseline.json— committed regression baseline (generated).llmwiki/wiki/sources/evals/*.md— goldens.harness/evals/wikieval-outputs.example.json— self-test candidate outputs.