nuke-test for model behavior (map: references/family-map.md — nuke-prompt's breaker findings arrive here as adversarial cases). A prompt edit is a code change whose gates nobody runs: someone reorders a sentence, output "feels better", and the regression ships. This skill maps what the prompt promises, builds an eval set, runs it before/after, and proves the evals themselves by perturbation — the mutation check's sibling: an eval no prompt-degradation can fail is vacuous and gets deleted.
Arguments
[mode] — light (default) | full | plan (preflight, print the plan block, STOP).
[target] — a prompt/instruction file, an agent directory, or changed (prompt-bearing files changed vs HEAD, default).
--ask — pause at the preflight plan for confirmation; default is no gate — the plan prints and the run starts (references/preflight.md).
Modes
Tier vocabulary and platform mechanics: references/model-tiers.md.
| light (default) | full | |
|---|---|---|
| Behavior mapper | session | session |
| Case writers | worker | worker |
| Judge design | mechanical asserts first; LLM-judge only where asserts cannot reach | same, LLM-judge rubrics reviewed at top |
| Perturbation check | 3–5 perturbations, riskiest behaviors | ≥1 per newly covered behavior |
| Runs per case | 1 (flag nondeterminism when seen) | 3 — a case must be stable or its flakiness is a finding |
| Cost stance | lean: riskiest behaviors first | costs do not matter; never shrink the map |
Mandates
- An eval that cannot fail is a defect. The perturbation check is the proof: degrade the prompt deliberately (drop a constraint line, invert an instruction, remove a few-shot) — the evals covering that behavior must fail. Survivors are coverage gaps; vacuous evals are deleted, never kept.
- Behaviors come from the prompt's contract — what it promises: output format, refusals, tone, task success, constraint adherence — read from the prompt text and its call sites, never invented.
- Mechanical asserts before LLM judges. Format checks, schema validation, string/regex presence, length bounds first; an LLM-judge rubric only for what asserts cannot reach, and the rubric itself states pass/fail criteria a second judge could apply identically.
- Never leave a perturbation applied. One at a time; revert verified before the next; the final tree contains only eval files and the baseline report.
- Model calls cost money — say so. The preflight estimate includes eval-run API calls (cases × runs × cycles), not just agent count.
- No
git add/git commit/git stash. Working tree left for review. - Never claim improvement without both numbers. Before/after pass rates from the same eval set, same model, same settings — or the claim is not made.
Artifacts
run_dir = .nuke/<YYYY-MM-DD>-<HHmmss>-eval-<slug>/ — fresh per run; on collision append -2, -3, ….
| File | Role |
|---|---|
plan.md |
Preflight output: target, prompt inventory, harness, wave, tiers, API-call estimate |
behaviors.md |
Numbered, risk-ranked promise map with coverage marks |
evals/ |
The eval set: cases + runner (kept in-repo only if the user confirms a destination in preflight) |
results.md |
Baseline scores, perturbation verdicts, before/after when a change is under test |
Pipeline
Phase 0 preflight (plan → apply) → Phase 1 promise map (session) → Phase 2 case writers (worker) → Phase 3 perturbation check → Phase 4 baseline/before-after + report (STOP)
└── surviving perturbation → Phase 2 for that behavior — cap 2 cycles ──┘
Phase 0 — Preflight
Read references/preflight.md and follow its plan-then-apply gate. Eval specifics:
- Resolve the target into a prompt inventory: every file whose content reaches a model (system prompts, instruction blocks, templates, few-shots, tool descriptions).
changedresolving to zero prompt-bearing files → STOP and ask. - Harness resolution: find the existing eval setup (eval scripts, test files calling models, promptfoo/braintrust-class configs). None → the plan proposes a minimal runner (one script, cases as data) and asks where it should live.
- Existing-eval inventory — the dedup baseline.
- Estimate agents AND model calls (Mandate 5). Print the plan block and apply it — gate only with
--ask.plan→ STOP.
Phase 1 — Promise map (session)
One mapper reads the prompt inventory and its call sites, and writes behaviors.md: numbered P-## promises, one observable behavior each, risk-ranked — task success on the critical path first, then refusal/safety behavior, then format contracts, then tone/style. Each marked covered (existing eval cited) or uncovered. Sources are the prompt text and call sites only (Mandate 2).
Phase 2 — Case writers (worker)
Writers in parallel, ≤5 promises each, disjoint case files. Every writer prompt embeds: its P-## entries verbatim · the harness contract · the assert-first rule (Mandate 3) · this rule-set:
- Golden cases: representative inputs with mechanically checkable expected properties — never exact-string-match on free prose.
- Adversarial cases per promise: the input most likely to break it (edge formats, injection attempts against tool-use prompts, instructions that conflict with the system prompt).
- One promise per case; the case name states the promise.
- A case that passes against the current prompt AND an obviously degraded one is not yet a case.
Phase 3 — Perturbation check
The breaker (session light / top full) proves the set: propose perturbations (X-##) — each targets one promise: drop its constraint line, invert it, delete the relevant few-shot, truncate the prompt. Apply one at a time to a COPY of the prompt wired into the harness; run the covering cases; killed (case fails) proves coverage, survived sends the promise back to Phase 2 with the perturbation quoted. Revert verified per Mandate 4. Cap 2 cycles; survivors after cycle 2 are reported gaps.
Phase 4 — Baseline / before-after + report (STOP)
- Run the full set against the current prompt → baseline scores into results.md. When the run's purpose was judging a specific prompt change: run against both versions, report per-promise pass rates side by side, verbatim.
- Report: promise map coverage, perturbation scorecard (killed/survived), vacuous evals deleted, baseline (or before/after) numbers, honest gaps, artifact paths, calibration line appended to
.nuke/calibration.log(format in references/preflight.md). - STOP. Regressions found in the prompt itself → recommend nuke-prompt for a disciplined rewrite; never rewrite here.
Orchestration notes
- Writers parallel on disjoint files; the breaker alone, perturbations strictly serial. Tiers per references/model-tiers.md; no per-agent overrides → every role inherits the session model.
- Platforms without a way to call the target model from the harness (no API key, no CLI) → build the case set anyway, mark
results: NOT RUN — no model access, and say exactly what command runs it later. Never fake scores.