SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslight-experiment-coding

light-experiment-coding

Testing & quality

View repository

Updated 10-16 06:23

About this skill

Experiment coding (stage 6)

The task is not to "write a notebook that runs", but to implement the upstream frozen plan line by line intofalsifiable, rerunnable, auditableexperiments. Priorities:

  1. do not change the research question;
  2. do not let evaluation information enter training;
  3. be able to truly rerun with a fixed environment and seed;
  4. retain enough raw evidence for result-analysis to recompute on its own;
  5. code cleanliness and speed are subordinate to the above constraints.

First read references/experiment-coding-resource-map.mdin full. For tool mechanics see references/tools.md, for TDD/debugging red flags see references/tdd_redflags.md and references/debug_protocol.md。

Entry point: frozen inputs

Before starting to write code, read and hash:

  • question / estimand；
  • each row of the experiment matrix and the fair-comparison constants;
  • pre-registration and provenance;
  • failure-tree report: the success/failure/inconclusive branch, guardrail/counter-metric, kill criterion, and amendment policy for each hypothesis;
  • data fixed revision、raw/curated SHA256、lineage、split ID、split_leakage evidence；
  • result-analysis's consumption contract for raw runs, predictions, metrics, failures, and provenance;
  • the current git commit and dirty state.

The primary outcome, comparison family, exclusion, and stopping are already frozen. If implementation proves the plan infeasible, return to research-plan with a minimal reproduction and impact, and stop for a human decision; do not silently drift by changing config defaults.

Implementation order

1. Establish a minimal runnable project

Prefer to copy assets/project-scaffold/：

  • uv.lock + pyproject.toml：uv sync --locked --extra dev；
  • configs/experiment.schema.json: machine-readable config for each matrix row;
  • experiment_contracts.py: minimal interfaces for data/model/metric/preprocessing;
  • reproducibility.py: runtime seed helper;
  • CI/pre-commit/debug assets.

Follow the existing repository framework and config format; do not introduce a paid IDE, cloud tracking, or private key for one experiment. MLflow/DVC are optional; ordinary local files must be able to complete the core loop. Clearly write unavailable resources as UNAVAILABLE:原因, do not pretend they pass.

2. Test first

Before implementing preprocessing/train/eval:

  1. write gold tests, verifying small answers that can be computed by hand;
  2. write property tests (Hypothesis), verifying invariants such as range, finiteness, symmetry/monotonicity;
  3. write metamorphic tests, verifying output relationships after permutation/equivalent transformations;
  4. write train-only-fit tests, recording that the transformer only receives the training fold;
  5. watch the new tests fail due to missing implementation or a real bug with your own eyes, then write the minimal implementation to make them pass.

For floating-point assertions use pytest.approx / assert_allclose(rtol, atol). First declare device, dtype, mixed precision, and tolerance; NaN/Inf fail by default. Do not test "random training will definitely reach some nice number"; test deterministic boundaries and recomputable facts.

3. Leakage-proof implementation

  • holdout: split first, then use only train fit/fit_transform, test only transform；
  • CV/tuning: put the preprocessor and model into a sklearn Pipeline, fit only the training fold per fold;
  • use group-aware split for entities such as patients/users/pastures, and do not cross train/test;
  • Target encoding, feature selection, PCA, and imputation are likewise fit only on the training fold;
  • Data artifact review directly reuses data-engineering split_leakage, without rebuilding.

review_gate only identifies static patterns, and will miss/false-positive; domain semantics still require manual verification. If metrics look abnormally good, check for leakage first.

4. Control randomness and numerical boundaries

Separate two purposes:

  • fixed_repro: fix one seed, and perform two independent runs under the same code/data/config/environment;
  • randomness_estimation: perform multiple runs according to a pre-registered seed list, to estimate algorithmic randomness.

Multiple seeds are not patient/entity sample size; never substitute the number of seeds for the power analysis n.

Cover according to actual use PYTHONHASHSEED(the current process must be set before startup),random, NumPy, framework RNG, CUDA, cuDNN, deterministic algorithms, DataLoader generator/worker. Record device/dtype/AMP, thread count, worker count, sort keys, and known non-deterministic operators. Changing the seed should be variable but traceable; same-seed consistency only supports the same-environment contract, and does not extrapolate to bitwise guarantees across release/platform/hardware.

5. Keep a raw bundle for every run

From templates/run_manifest.template.json generate light.run_manifest.v3; for explanation see templates/run_manifest.md。 termination only explains why the process stopped,completion records whether the matrix row oracle actually passed; timeout / max iterations / cancel must never be passed off as completion.

Each matrix row × config × seed × attempt gets an independent directory, saving at least:

  • snapshot after config schema validation;
  • code commit/dirty/diff and code file hashes;
  • data/input SHA256、source revision、split ID；
  • environment (Python/packages/OS/CPU/GPU/CUDA/threads);
  • argv array, stdout, stderr, exit code;
  • raw metrics per fold/per entity, patient/entity-level predictions;
  • test evidence；
  • failure artifact for failed runs.

Failed runs are not deleted and not overwritten as successful. Submitting only a summary CSV is prohibited; result-analysis must be able to recompute from raw evidence.

Gating and checkpoint

Actually run in this order and record exit codes:

python scripts/experiment_execution_contract.py --spec experiment_execution_contract.json \
  --report execution_contract_findings.json --json-out execution_contract_report.json
python scripts/review_gate.py src/ --json
python scripts/seed_audit.py src/train.py src/reproducibility.py
python scripts/repro_gate.py --spec repro_spec.json --report repro_findings.json
python scripts/run_artifact_check.py --manifest runs/EXP-01/run-a/manifest.json
python scripts/run_artifact_check.py --compare \
  runs/EXP-01/run-a/manifest.json runs/EXP-01/run-b/manifest.json
python ../light-orchestrator/scripts/run_checkpoint.py \
  --file .light/passport.yaml --stage 6 --findings repro_findings.json --write --ts <ISO-8601>

experiment_execution_contract.py consumes light.experiment_execution_contract.v1(template see templates/experiment-execution-contract.example.json, intentionally fail-closed): check as_of, frozen scope/evaluator/budget, failure-tree handoff, matrix/DAG, run status/termination/failure class, partial checkpoint/resume command, environment and cache provenance, repro level layering, remote/paid execution authorization.timeout/OOM/preempted/max-iterations must not be written as completed；frozen_at and remote authorization approved_at must not come from the future;decision=NOT_READY/UNKNOWN itself blocks progress; a run must be bound to a row in the frozen matrix_rows , and each row must be bound to failure_tree_refs(hypothesis_ids, branch_action_ids, applicable guardrail_ids); adding a new experiment row requires first going back to research-plan for revision; if a completed run requires a guardrail upstream,completion must leave guardrail_evidence_artifacts, otherwise it must not be submitted to result-analysis; actual walltime/cost/compute_units and remote estimated cost must not silently exceed the frozen budget, and exceeding the limit requires budget override authorization;PARTIAL/RESUMABLE must have a checkpoint SHA and resume command; requesting CLEAN_ENV_RERUN/CROSS_PLATFORM/INDEPENDENT_REIMPLEMENTATION must have corresponding evidence; remote/paid/HPC runs must not user_authorization=APPROVED before RUN_READY。

The canonical critical items of stage 6 are only:

  • leakage；
  • reproducible。

float/security is warn at the research checkpoint per spec;review_gate Can serve as an independent delivery blocking gate, and must not be used to expand the checkpoint critical surface. If the stage-6 gate fails, fix it in this stage,Do not fabricate ROUTES[6] outgoing edges。

Two runs with the same fixed seed must be independent run dirs;run_artifact_check.py first verify matrix/config/env/code/input identity and started_at/ended_at timeline (with time zone, not reversed, not from the future), then compare predictions/raw_metrics hashes. Even if the static seed/leakage gate + same-seed pair both pass, it is still only one bounded piece of evidence.

Hand off to result-analysis and 7→6

Deliver all completed/failed runs, coverage, manifests, predictions, raw metrics, logs, test evidence, and canonical findings. Do not selectively drop runs or write conclusions first on behalf of result-analysis at stage 6.

Only when result-analysis producesa real implementation bug or non-reproducible root causemay the orchestrator:

python ../light-orchestrator/scripts/reroute.py \
  --findings result_findings.json --stage 7 --passport .light/passport.yaml

reroute only gives suggestions.Stop before creating the 7→6 back-edge and let the user decide; only after user confirmation call passport add-back-edge, and bring back the failed command, expected vs actual, and artifact pointers to fix the root cause. Statistical non-significance, small effect, or infeasible plan does not automatically equal an implementation bug.

Non-negotiable

  1. Do not fit scaler/encoder/imputer/PCA/feature selector on the full dataset before split/CV.
  2. Do not set only one framework seed and claim reproducibility.
  3. Do not use floating-point == assertions, and do not silently swallow NaN/Inf.
  4. Do not mix the scientific purposes of a fixed seed and multiple seeds.
  5. Do not delete failed runs, keep only summary numbers, or omit stdout/stderr/predictions/raw metrics/provenance.
  6. Do not write a static scan as “proof of no leakage,” and do not write two consistent same-seed runs as “absolute reproducibility across hardware.”
  7. Do not change the frozen research design; if infeasible, go back to research-plan and let a human decide.
  8. Do not let stage 6 create outgoing edges by itself; 7→6 is only suggested by a real downstream root cause, and the edge is created only after user approval.

Completion criteria

  • Every matrix row has a schema-valid config and a stable run ID;
  • Has it been run experiment_execution_contract.py ? Are scope/evaluator/budget, DAG, run status, failure class, resume, resource cost, repro level, and remote authorization all closed?
  • Is the execution contract bound to .light/failure_tree_report.json locator/hash/status? Does every matrix row have failure_tree_refs? Did completed runs leave guardrail evidence?
  • as_of、frozen_at, remote approved_at、run started_at/ended_at all real times that have occurred? No future pre-filling or end earlier than start, right?
  • run's matrix_row_id come from the frozen experiment matrix? When actual walltime/cost/compute_units or remote estimated cost exceeds budget, is there budget override authorization?
  • gold/property/metamorphic/train-only-fit tests red first, then green;
  • lock, interface, device/dtype/AMP/NaN/Inf/tolerance boundaries are clear;
  • review/seed/repro gates and stage-6 checkpoint pass;
  • fixed same-seed two-run predictions/raw_metrics hashes match;
  • changes from switching seeds are traceable via manifest;
  • every run's config/env/code/input/log/raw metric/prediction/failure evidence is complete and hash-verifiable;
  • result-analysis can be consumed from the bundle alone, with no plan drift or unauthorized back-edge occurring.

Name-reality alignment

TDD, seed, security scanning, and reproduction experiments are not unique to Light. True peers in R1 already generally have config-first, smoke, immutable run, lineage, and keep/discard. The actual increments here are:

  1. code-level holdout/CV leakage;
  2. framework-aware seed audit;
  3. reuse of data artifacts split_leakage；
  4. frozen execution contract (scope/evaluator/budget/DAG/status/resume/repro level/remote authorization); Round 3 then adds run→frozen matrix row identity binding and actual/estimated resource cost budget overrun gates, to prevent new experiment rows or paid scale-out runs from bypassing upstream approval; Round 3 further adds failure-tree handoff and guardrail evidence, so that research-plan failure/no-conclusion/kill criteria are not lost on the execution side;
  5. hash-verified raw run bundle + same-seed pair；
  6. canonical findings/stage-6 checkpoint + user-controlled 7→6.

Honest shortcomings: no MLflow-style query UI, DVC remote, GPU scheduler, repository semantic index, or autonomous experiment search; static gates still have boundaries.