ue5-deterministic-sim-tests

Deterministic Simulation and Evidence

"The code is written" is not evidence. Neither is "it compiles", "it looks right", or a screenshot of the editor. For agent-driven development this matters more than usual, because the agent cannot see the game and the human is not watching every run.

The cheapest evidence in a vehicle-combat game is a seeded headless match: run twenty full AI matches in seconds, with no rendering, and assert every one terminated with a legal winner. That single test catches AI stalls, unclosable endgames, scoring bugs, and match-flow regressions at once — and it costs nothing to re-run on every change.

Establish the determinism contract first

Determinism is not a feature you add later. It is a set of constraints you either adopt at the start or pay dearly to retrofit.

1. Fixed logic timestep. Gameplay state advances on a fixed step, accumulated from real time. Rendering interpolates. Anything gameplay-authoritative that integrates with a variable DeltaTime is non-deterministic by construction.
2. Seeded streams, one per subsystem, all derived from a single match seed. Never a global RNG. See references/determinism-rules.md.
3. No gameplay reads from presentation. Physics-driven visual motion, particle state, and animation must not influence outcomes. This is the rule that makes headless running possible at all.
4. No frame-rate-dependent smoothing in gameplay code — time constants, never per-frame lerp factors.
5. Iteration order must be stable. Iterating a hash map whose order varies between runs silently breaks determinism in a way that is very hard to find.
6. Simulation must link without rendering. Enforced by module boundaries — see ue5-project-context.
7. Decide the tolerance. Bit-exact is achievable if you are careful with floating point; "same winner, same seed" is much cheaper and usually sufficient. Pick one and state it, because it determines how strict every assertion must be.

Load only what applies:

• references/determinism-rules.md — the constraints, the common violations, and how each one presents.
• references/headless-harness.md — building the no-rendering match runner and what to log.
• references/evidence-artifacts.md — what to save so a claim is checkable later, and what "done" means.

Required answer format

Return:

1. Determinism contract — timestep, tolerance level, and the stream list.
2. Violations found — every place gameplay reads presentation, uses a global RNG, or uses variable-step integration.
3. Harness description — how a match runs headlessly, and how long one takes.
4. Test inventory — what is asserted, at what level (unit, functional, batch).
5. Batch results — matches run, outcomes, durations, failures.
6. Artifacts — paths to logs, CSV dumps, replays, screenshots, and performance baselines.
7. Known gaps — what is not covered by a test, stated plainly rather than omitted.

Hard rules

• A test that has never failed has never been checked. Break the thing it tests, confirm it fails, restore. An assertion that cannot fail is decoration, and this class of test is especially prone to it because a headless match "passing" can mean it never ran.
• Never assert on wall-clock timing in a simulation test. Assert on simulated time.
• Seeds must be explicit and logged. A failure you cannot reproduce is an anecdote. Every batch run logs its seed list, and the log is the artifact.
• Different seeds must produce different outcomes. If all twenty matches end identically, the seed is not reaching the systems that matter — a passing batch that proves nothing.
• Evidence is a file, not a sentence. "Tests pass" in a report without a path to the report is not evidence.
• Never report an untried command. A runbook containing a plausible command that was never executed is worse than an empty runbook, because it will be trusted.
• Report the failures too. A run summary that lists only successes is an unreliable narrator, and the next agent will act on it.
• Performance numbers come from packaged builds with the hardware, settings, and scenario recorded. Editor numbers are not comparable to anything, including other editor numbers.

Verification

• Repeat run. Same seed twice → identical result at the stated tolerance.
• Frame-rate independence. Same seed at 30 and 120 fps caps → identical result. This is the test that catches variable-step integration and per-frame lerp.
• Seed spread. 20 seeds → a distribution of outcomes, not one repeated outcome.
• Termination. Every headless match ends with a legal result inside the hard timeout.
• Mutation check. Deliberately introduce a bug (disable AI target selection, break the lead solver) and confirm the batch fails. Restore.
• Cold-start check. A fresh agent with no conversation history runs the documented commands from the repository and reproduces the reported results. This is the acceptance test for the whole evidence system.

Scope

Not covered: networked determinism and lockstep, replay compression formats, cross-platform floating-point determinism (hard, and rarely needed for single-player), CI infrastructure setup, and performance profiling method (that belongs to the platform skill).