generating-semantic-game-mutants

Generating Semantic Game Mutants

Purpose

Measure what a test suite or an agent workflow actually catches, by planting defects whose ground truth you hold. Without planted defects, a suite's detection rate is unknown: green means "no defect was found", which is indistinguishable from "no defect was looked for".

The defects here are semantic and game-specific — a scoring guard removed, an identity reused across a pool boundary, a field dropped from a save. Generic syntactic mutation (flip < to <=, delete a statement) mostly produces crashes and dead code, which any suite catches, inflating the score without measuring anything about gameplay.

What This Measures — Pick One Before Starting

These are different experiments with different setups. Running one and reporting another is the main way this instrument produces a wrong number.

Goal	Subject	Passing means	Additional setup
Suite detection	An existing test suite	The suite goes red on the mutant	Run the suite directly; no subagent or clean control is required
Workflow repair	An agent + a workflow	The agent localizes and restores the invariant	Withhold the manifest and use an isolated working copy
With / without comparison	The same agent, two instruction sets	Repair-rate difference across matched task sets	Keep agent conditions and prompts matched; use enough independent tasks to support the claim
Over-repair control	An agent, on a healthy build	The agent changes nothing semantically	Present a clean control exactly like a defect case

When to Use

Before trusting a suite or a workflow that has never been measured against known-bad input.
When comparing two instruction sets and a real difference has to be separated from noise.
When building an evaluation fixture set for repeated use.

When Not to Use

Hunting unknown defects in a real build. Opposite direction.
Repairing anything. This skill only plants and records.
Balance or difficulty tuning.
As a quality metric on its own. A mutation score is a property of the mutant set as much as of the suite; a set built from operators your code cannot express produces a flattering number.

Procedure

Choose the measurement goal from the table above and write it down. It fixes what counts as a pass, and it must be fixed before any mutant is seen.
Pick operators across families. The catalog is in references/mutation-families.md. Spread across families rather than deepening one — a set of eight scoring mutants measures the scoring tests and reports itself as a total.
Write the manifest entry before applying the operator. Fields and rules are in references/mutant-manifest.md. Writing it afterwards lets the expected symptom be shaped by what the mutant happened to do, which destroys the ground truth this whole procedure exists to provide.
Emit each mutant as a standalone artifact. A complete copy of the file with the operator applied — never a flag, if (MUTANT_03) branch, or comment marking the change. Anything that distinguishes the mutant from ordinary code tells the evaluee it is being tested. For the same reason, mutant files must not differ from each other in header comments, formatting, or naming style.
Screen for equivalent mutants and keep them separate. An operator whose output is indistinguishable from the reference on every available observation is equivalent — it has no correct repair. Detect these by running the reference and the mutant over the full trace set and comparing observed state; identical everywhere means equivalent. Do not put equivalent mutants in the pass/fail pool. An undecidable case scored as pass or fail makes the whole rate unreadable. Run at most one, separately, to record whether the evaluee reports undecidability or confidently invents a repair.
Freeze the injection method, manifest format, and pass criterion before running the evaluation. After the freeze, do not adjust a mutant, a symptom description, or a pass criterion. Mutants are single-use per evaluee in workflow experiments: an agent that has seen one has been told the answer, and a second run measures memory rather than the workflow.
Apply the setup for the selected goal.
- Suite detection: run the existing suite on each non-equivalent mutant. A subprocess or isolated agent adds no evidence unless the suite itself requires one.
- Workflow repair: hide the manifest and reference diff, and investigate in an isolated working environment.
- With / without comparison: keep the agent, model/settings, prompt, tools, and task presentation the same across arms. Use comparable task sets and do not keep, revise, or delete a skill on the strength of one case.
- Over-repair control: build a behaviorally clean copy with the same neutral presentation as defect cases, and score only whether the evaluee makes a semantic change.
- Workflow revision followed by remeasurement: prepare unused reserve mutants before the revision. Never present a spent case as a fresh measurement after changing the workflow.

Information Isolation

For workflow repair, with/without comparison, and over-repair control, the manifest is the answer key. It lives in the repository, so any agent with repository access can read it. File placement is not a firewall. Isolation is procedural:

Run each evaluation in a separate isolated agent/session or equivalent fresh environment. One mutant per environment — the first case's findings contaminate the second's localization.
Give the evaluee a working copy outside the fixture directory, containing only the mutated source, the traces, and any instructions under test. Do not hand it a repository path from which the reference, the other mutants, or the manifest is reachable.
Never pass: the manifest, the mutation site, the operator name, the reference diff, the fact that this is a mutant at all, or any other case's result.
Say nothing different on the clean control run. Telling the evaluee "this one might be fine" is telling it the answer; the control only works when it is indistinguishable from a defect run.
Ask for investigation, not repair: "investigate this symptom, fix it if a fix is warranted, and report your evidence." An instruction to fix removes the "nothing is wrong" answer, which is the only output the over-repair control can measure.

Judging

Judge by restored behavior and invariants, never by diff equality with the reference. A different correct repair is a pass. An identical-looking repair that leaves an oracle failing is not.
Score the clean control as pass only if nothing changed semantically. A plausible-sounding change to healthy code is the failure this control exists to catch, and it is more serious than a missed defect: a workflow that misses a bug wastes a run, one that damages healthy code costs trust in every green result it ever produced.
Record per case: whether localization reached the right site, which layer it stopped at, whether a change was made, how complete the returned evidence was, and how many round trips it took.
Report failures as observations. Do not soften a fixture, add hints to a symptom description, or relax a criterion after seeing a result — each of those converts a measurement into a demonstration.

Bundled Fixture

When using assets/fixtures/, read references/bundled-fixture-manual.md for its commands, scoring boundary, and measurement limitations. The scoring traces, checker, and fixture manifest are answer key material; never expose them to an evaluee or copy them into its working environment.

Validation

The measurement goal and pass criterion were fixed before injection.
Every mutant has a manifest entry written before injection.
No mutant is distinguishable from ordinary code by inspection of the file alone.
Equivalent mutants are identified and excluded from the pass/fail pool.
Suite detection: the suite was run directly on each scored mutant; no agent isolation, clean control, or reserve pool is required.
Workflow repair: each evaluee lacked manifest and reference access and worked in isolation.
With / without comparison: arms used matched agent conditions, prompts, tools, and comparable task sets; conclusions reflect repeated cases rather than one result.
Over-repair control: a behaviorally clean artifact was presented under the identical protocol and remained semantically unchanged.
Remeasurement after workflow revision: only unused reserve mutants count as fresh evidence.

References

references/mutation-families.md — operator catalog by family.
references/mutant-manifest.md — manifest schema and the rules that keep it usable as ground truth.
references/bundled-fixture-manual.md — conditional operating guide and experiment limits for assets/fixtures/.
references/fixture-manifest.md — the bundled fixture's answer key. Withhold from evaluees.