light-research-plan

Research Plan and Experimental Execution Planning

Prerequisites

Execute only for ideas already approved by m04. Before starting, confirm that data (m02) and methods (db03) are ready.

Plan Content (write each item concretely, no empty talk)

1. Research Objectives & Core Questions: Falsifiable hypotheses H1/H2…. Confirmatory research (with hypotheses to be tested in advance) may bepreregistered, locking down hypotheses/primary metrics/analysis plan to prevent HARKing/p-hacking; exploratory analyses must be honestly distinguished in the paper and not packaged as confirmatory (for the minimal workflow and field mapping, see the "Preregistration" section in references; the sample size rationale reuses power analysis).
2. Technical Approach: Overall framework diagram (hand off to m09/m11), key modules and data flow.
3. Data: Sources, processing pipeline (cite the m02 pipeline), splitting method, statistics.
4. Model/Method: Algorithm workflow, formulas, pseudocode, complexity.
5. Experimental Design: Write every experiment entry completely according to the four elements of EXP-Bench — research question (which hypothesis it corresponds to), design (independent/dependent/control variables + dataset + baseline + metrics), implementation (code + run configuration), conclusion determination (what results are used to answer that hypothesis). "Design" and "conclusion" are the easiest to go off track, so focus on checking them.
   • Main experiments: tasks, datasets (from db04: scale/splits/license/known issues), baselines (from db03), evaluation metrics.
   • Ablation experiments: remove innovative components one by one to demonstrate the source of contribution.
   • Comparative experiments: fair comparison with SOTA (same data/same settings).
   • Parameter sensitivity: grid over key hyperparameters + trend analysis. Use Hydra multirun (-m lr=0.01,0.1 model=a,b Cartesian product) or W&B Sweeps (method=grid/random/bayes + metric goal + parameters) to systematically sweep parameters.
   • Generalization tests: cross-dataset/cross-domain/cross-scale.
   • Robustness: noise/adversarial/missing/distribution shift.
   • Statistical significance:≥5 random seeds; under compute constraints ≥3 and must be explicitly noted in the m06 report(same standard as the m06 mandatory checklist); for inference use statsmodels (OLS/Logit, check the p-values and 95% confidence intervals in summary, remember add_constant) or scipy (t-test/Wilcoxon); when uncertainty quantification is needed, use PyMC (check r_hat≈1, sufficient ESS). Report mean±standard deviation + error bars.Do a power analysis first for seeds/sample size: use TTestIndPower().solve_power(effect_size, power=0.8, alpha=0.05) to back out the minimum number of repetitions — a small number of seeds is only enough to detect large effects (d≈0.8 requires about 26 per group), a medium effect (d≈0.5) requires about 64 per group (for numerical verification by actual runs, see the "Statistical Power and Sample Size/Seed Number Planning" section in references); don't wait until the runs are finished to discover that differences cannot be detected.
   • Leakage prevention: all preprocessing that depends on training-set statistics (standardization/imputation/encoding) must be encapsulated in an sklearn Pipeline + ColumnTransformer, then passed to cross-validation; fitting on the full dataset first is prohibited.
6. Visualization Plan: Which charts to produce (submit to m09).
7. Timeline: Milestones + Gantt, aligned with the deadline.
8. Risks & Alternatives: Each risk paired with a plan B.
9. Compute/Cost Budget: Once the idea is approved, the first step must be doing the math—estimate GPU hours × number of GPUs × unit price per experiment, and aggregate against the budget cap (the template experiment_matrix.md already includes a budget table). Record the source + date for unit prices (aligned with db08); no relying on memory. Sweeping parameters × multiple seeds is a budget amplifier; if over budget, cut scope. Follows on from m04 rubric dimension 7.
10. Expected Outcomes: Paper tier/competition/patent/software copyright.

Reproducing an Existing Paper (Reproduction Is Also a Research Plan)

Reproducing the results of a paper is a legitimate research plan (verification/teaching/as your own baseline). It shares the reproducible planning with designing from scratch, plus five steps for benchmarking against the original text (see the "Protocol for Reproducing an Existing Paper" section in references):

1. Define the reproduction goal: Anchor which number in which table/figure (e.g., Table 2 top-1 76.5%); do not state an unverifiable goal like "reproduce this paper."
2. Asset inventory: Availability of the four categories—code/weights/data/config—determines reproduction difficulty and room for attribution.
3. Deviation budget: Define in advance "how much difference counts as success" (e.g., within ±0.5% of the original); without a preset, you will end up in endless disputes.
4. Reproduction log: Record each run as {what was changed/the number obtained/difference from the target/next hypothesis}, changing only one variable at a time.
5. Honest attribution of failure (three-way split): When the deviation budget is not met, attribute it to implementation differences / data differences / problems in the original paper; do not vaguely say "it didn't reproduce." Pointing to "problems in the original paper" requires cross-validation from multiple sources and going through research-ethics; do not rashly make public accusations. For reproduction code and debugging, go to a03(light-backend-coding); when reproduction is used as a baseline, results go back to m06.

Reproducible Planning (Mandatory)

Implement item by item, giving concrete configurations rather than tool names:

• Environment: Record OS/driver/CUDA; lock dependency versions (requirements.txt with pinned versions / environment.yml / lockfile).
• Directory scaffold: Follow the Cookiecutter Data Science layout—data/{raw,interim,processed,external}(raw is read-only, not modified),src/(reusable logic sinks down; notebooks do not hold core logic),models/、reports/figures/、Makefile、README。
• Configuration management: Hydra layered configuration (model and dataset groups under conf/ + defaults list composition), command line can override lr=0.1, and the run automatically saves the final composed configuration.
• Data/model versioning：DVC——dvc add tracks large files (git stores only .dvc pointers),dvc.yaml defines stages (cmd/deps/params/outs/metrics),dvc repro incremental reproduction,dvc.lock locks hashes;dvc exp run/show comparison experiments.
• Pipeline: Snakemake rule (input/output/params/wildcards) automatically derives the DAG + incremental reruns; each rule uses conda:/container: to lock the environment.
• Experiment log：MLflow（set_experiment→start_run→log_param/log_metric(step=)/log_artifact, or autolog()) or W&B (init(config=)→log(), Artifacts manage lineage; for sensitive data use offline mode to avoid external transmission).
• Fixed items: Random seeds, data splits, hyperparameters, training strategy, result file naming conventions, run commands.

Execution discipline (implementation and verification, drawing on engineering practice)

• Plan before executing: The plan is divided into phases, each stating verifiable success criteria; during execution, advance one phase at a time, check off against the criteria before moving to the next phase, and write deviations back to the decision_log.
• TDD for critical code: For critical functions such as data processing, metric computation, and statistical testing, write the test first (provide a "known input → known output" gold-standard case), watch it fail, then implement; beware of "all tests green but the implementation is wrong."
• Systematic debugging: For bugs, follow the order of reproduce → read the full error → bisect to locate the root cause → verify the hypothesis → after fixing, return to the reproduction to confirm; if the same approach fails twice, stop and find the root cause and change tactics, do not keep patching.
• Verify before completion: Before declaring "done," go through a checklist—build passes, relevant tests pass, outputs match the success criteria item by item, temporary files cleaned up.

Outputs

1. Complete plan document (all of the above).Standard artifact:PROJECT_PLAN.md(hand over to a03/m06, artifact contract see CONVENTIONS §6.1).
2. Experiment matrix table (experiment × dataset × metric × status).Standard artifact:experiments/experiment_matrix.md(underscore naming, consistent with the contract).
3. Reproducibility checklist.
4. Hand over implementation code to a03, create directories in a06, register milestones in a02 and milestones to db09.

Ready-made templates are in the same directory templates/(research-plan.md / experiment_matrix.md / reproducibility-checklist.md). After filling in the experiment matrix, you can run python scripts/plan_lint.py --file experiments/experiment_matrix.md to self-check whether each row has all four elements (hypothesis/variables/metrics/stopping conditions).

Interfaces

Plan handed to a03 for implementation → experiments completed → m06 result-analysis; plan changes written back to db09 decision_log. Derived data back-edge: The derived evaluation sets required for robustness/generalization/sensitivity in the experiment matrix (noise-added/missing/cross-domain/parameter sweep) are sent back to m02 (light-data-engineering) as derived data specifications for construction, and the resulting dataset + dataset_card are backfilled into db04.

---

For tool details (real endpoints/parameters/commands, known pitfalls), see the same directory references.md。