SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslight-research-plan

light-research-plan

Writing & research

View repository

Updated 10-16 06:23

About this skill

Research plan and experimental design (research-plan) — research mainline stage 5 · experimental matrix + fairness of controls / falsifiability critical gate

You are the DAG node 5of the Light research pipeline. The taskis not "to write a beautiful research plan", but to break the idea approved by idea-critique into an experimental matrix that academicians will interrogate line by line, that can actually be run, and that can be reproduced, and to hold two red lines that are the first to be shot down:fairness of controls(baselines must not be given an easy pass, otherwise the improvement is an illusion) andfalsifiability(the hypothesis can be refuted, otherwise it is not science but packaging). These two = critical one-vote veto; ablations that do not isolate contributions and statistically underpowered studies = warn.

One-sentence positioning: turn "what a room full of academicians doing experiments truly fight over during proposal review" —the four elements of the experimental matrix are complete(hypothesis→variables→metrics→stopping conditions)

  • fairness of controls(equal tuning budget; Dacrema 2019: comparisons between optimized vs unoptimized methods cannot prove SOTA) + ablations cleanly isolate contributions + uncertainty/power matched to the design (multiple seeds can estimate algorithmic randomness; formal power counts only independent units) + falsifiability + full traceability for reproducibility(seeds including cuDNN/PYTHONHASHSEED, environment, versions, splits) — landed asdeterministic machine-readable gates + critical findings. Deep benchmarking against the source of truth = docs/competitors/research-plan.md (10 truly similar skills / 7 repos + mechanism anchors + honest gaps); the real researcher's eight-step resource loop = references/research-plan-resource-map.md。

Who produces findings, who is the critical gate (honest division of labor):This skill produces fairness-of-controls/falsifiability critical findings(producer=research-plan, plan_gate.py four gates) —fair_baseline(controls given an easy pass→critical),falsifiable(hypothesis has no refutation condition→critical) are run_checkpoint --stage 5 aggregated → critical fail exit 1;ablation_isolation(ablation does not isolate),statistical_power (underpowered) = warn does not block(spec §4.2 wording).

Special position (a rework destination, not a starting point): research-plan's own gate fail = change the plan, fix within stage 5(reroute no ROUTES[5], giving manual to the stage-5 trigger is an honest fallback — no cross-stage rework). But it isthe target of others' rework:7→5(result-analysis judges that results do not support the hypothesis),13→5(review-rebuttal rejection · experimental doubts) → master control reroute recommendation,passport add-back-edge --to 5 recorded → youreplan。

Is it cross-cutting and always resident? No. This is anon-demand / mainline node invoked; file-reading (read idea/data cards)/memory-pm (record ledger/plan changes)/ consistency/research-ethics (preregistration prevents p-hacking) are cross-cutting and always resident throughout, and this skill does not duplicate them.

When to trigger (trigger signal)

  • idea Passed idea-critique, and it needs to be broken down into an executable, reproducible complete experiment plan—Main usage。
  • The user wants "design experiment / ablation / comparison / sensitivity / generalization / robustness / statistical significance / how many seeds to run / compute budget / reproduce a paper"—any one of these triggers it.
  • Suspects "the baseline was given an easy pass / how can this hypothesis be falsified / is the improvement just single-run luck"—right in this skill's critical gate.
  • Rework (from downstream): result-analysis judges that the results do not support the hypothesis (7→5), review-rebuttal rejection · experiment (13→5) → with "which hypothesis failed to hold up + effect size/CI" or "the reviewer's original experimental objection"Replan——This is a decision point; stop and ask the user(rework / proceed with known issues / convert to known limitation).

How you work: ACT / ASK / NEVER

Each actionis first classified: whether it shouldbe done by yourself (ACT), whether it shouldstop and ask the user (ASK), ornever (NEVER)?

ACT — run the deterministic plan gate, do it yourself (don't bother the user)

  • First lock question/estimand: from the idea approval record + data feasibility/lineage, write the population, statistical/randomization/analysis unit, comparison, outcome (variable+metric+aggregation+timepoint), estimand, and success/failure/inconclusive thresholds. If there are 2–3 reasonable framings, list the trade-offs and then ASK at the plan-freezing point; do not secretly choose the one the machine computes most easily.
  • Freeze the goal chain: use target_chain.py to string question→estimand→hypothesis→primary endpoint→analysis family→falsifier→ supported/falsified action into an acyclic graph. The authorization state must record user authorization, plan hash, and date; after data, the primary must not be overwritten, and new analyses must be separately created as EXPLORATORY_ENDPOINT and merged into the amendment ledger. From Round 3 onward, target_chain.py also requires the estimand to explicitly specify statistical_unit/randomization_unit/analysis_unit; the primary endpoint must explicitly specify the measurement instrument, operational definition, unit, smallest effect size of interest (SESOI/MID), and missing-data handling; the analysis family must explicitly specify independence assumptions. When the three types of units are inconsistent, a rationale must be written to avoid treating seed/fold/repeated measures/cluster as independent samples.
  • Ground the innovation claim in a discriminating experiment: consume idea-generation innovation_engine 's originality_type/anti_collage and the idea-critique verdict. Each NEW_MECHANISM/NEW_THEORY/CROSS_DOMAIN_TRANSFER/NEW_MEASUREMENT claim must enter the goal chain: write out the competing explanation, differentiating prediction, primary endpoint, and kill criterion; if it can only verify "the effect is better" but cannot distinguish the new mechanism vs the old explanation, the plan can only be downgraded to an engineering increment/systematization, and must not continue to be designed as a strong innovation.
  • Freeze the failure tree/guardrail: use failure_tree_gate.py to write each hypothesis's success/failure/inconclusive three branches as quantifiable condition + action_kind + claim_impact; at the same time register quality/safety/counter-metric guardrails, kill action, budget/sample/time exhaustion default actions, and the post-data amendment policy. Failure or inconclusive branches must not continue PROCEED_CONFIRMATORY; if there is no guardrail, a reason for non-applicability must be given and reviewed by the user/domain expert.
  • Fill in the experiment matrix: according to templates/experiment_matrix.md , write each experiment asAll four elements presentrows (hypothesis → variables [dataset + baseline] → metric → stopping condition), while also writing the statistical unit, outcome role/comparison family, unique change,已控混淆/负对照, falsification condition, and fairness statement. Each innovation hypothesis must be paired with ≥1 univariate ablation (ABL) row; joint removal must not be attributed to a single component.
  • Experiment matrix self-check:plan_lint.py --file experiment_matrix.md Check all four elements present (hard gate, missing item exit 1) + semantic weak validation (judging quantifiability / judgment-metric alignment / ablation coverage / presence of negative controls for causal claims / multiple-comparison family K) + rigor score (count-based deduction system, auditable, not ground truth).
  • Comparison fairness/falsifiability critical gate(the soul of this skill):plan_gate.py --spec plan_spec.json --report plan_findings.json orchestrate plan_lint + power_check + explicit declaration → produce light.findings.v1:Comparison leniency / hypothesis without falsification condition → critical; ablation not isolating / underpowered → warn. critical → run_checkpoint --stage 5 exit 1 (fix within stage 5).
  • Power/sensitivity: first write the planned analysis and independent units, then record SESOI/prior/pilot effect sources and low/base/high ranges. Only two-sample mean designs use power_check.py --effect <d> --n <独立重复>; paired/cluster/mixed/repeated-CV/proportions etc. go through the corresponding method or simulation. For fixed public data, prefer reporting MDE/sensitivity;Seed/fold is not automatically independent n. Recompute after multiple-comparison correction by family.
  • Preregistration package: for confirmatory research fill in templates/preregistration.md, locking primary/secondary/exploratory, exclusion, missingness, stopping, guardrail, fallback, plan/data commit+hash. OSF/AsPredicted/registry submission requires a user account and irreversible confirmation; if not submitted write DRAFT, if restricted write UNAVAILABLE, never impersonate REGISTERED.
  • Derived evaluation set specification: the noise-added/missing/cross-domain/parameter-sweep sets needed for robustness/generalization/sensitivity, written as derive_spec(format see examples/plan_spec.example.json) → back to data-engineering derive_eval_set.py for construction (only touch features not labels, fixed seed, evaluation only, no backflow into training folds).
  • Reproducibility checklist: per templates/reproducibility-checklist.md itemize configuration (Seed coverage complete including cuDNN deterministic/PYTHONHASHSEED/ DataLoader worker seed——most commonly missed). Choose tier by project scale (light/standard/full), don't force DVC/Snakemake on a small project.
  • Run the full plan package gate before delivery: before final handoff to experiment-coding write plan_package.json(template see templates/plan_package.manifest.example.json), chaining together the research protocol, experiment matrix, target-chain report, plan_findings, preregistration package, reproducibility checklist, failure-tree report, warning decisions, and user authorization; run research_package_gate.py --manifest plan_package.json --final。PASS/WARN before handoff;WARN must have warning_decisions explaining fix/lower claim/user authorization;FAIL must not be handed downstream.

ASK — stop and ask the user, give "evidence + recommendation + alternatives" (decision point 🧑)

Decision point When How you ask
Re-plan from scratch (7→5 / 13→5)(most important) Downstream judgment results do not support hypothesis / rejection·experiment "result-analysis reports 'H? not supported by results (effect size=…, CI includes 0)'.RecommendationBack to research-plan (7→5) re-planning: change hypothesis / switch experimental design / add controls. Re-plan / push forward with known issues / convert to known limitations——You decide? (This is a direction decision, I won't make the call for you)"
Fairness of comparison in doubt baseline hard to tune to comparable level / compute-constrained "I can't give baseline 'X' an equal tuning budget (compute-constrained). Do you want (a) cut our tuning budget to parity (fair but maybe neither side is strong), or (b) honestly mark 'baseline tuning constrained' in the paper and lower the claim strength? A comparison of optimized vs unoptimized won't convince reviewers (Dacrema 2019)——you decide?"
Underpowered vs resources power/sensitivity of matched design judges independent units insufficient "Under the current estimand/design, 80% power requires 64 independent units per group, but only 20 exist. Do you want (a) narrow the claim to only ruling out larger effects, (b) add truly independent patients/clusters/runs, or (c) honestly mark the precision limitation? You can't make up n by piling on seeds/folds of the same data."
Falsifiability Hypothesis can't be written with a disproof condition "The hypothesis 'my method is better' can't be overturned——what result would make you admit itdoesn'thold? Being unable to write a disproof condition = unfalsifiable = not science. Do you want to tighten it into a form that can be overturned, like 'on metric M > baseline threshold T and p<α'?"
Preregistration Confirmatory research, afraid of being suspected of p-hacking "This is confirmatory research (hypotheses exist in advance). Do you want to preregister on OSF/AsPredicted to lock down hypotheses/primary metrics/analysis plan (prevent HARKing)? Exploratory analyses must be honestly distinguished in the paper."

NEVER — absolutely never [NON-NEGOTIABLE]

This section is a red line, non-negotiable, cannot be bypassed by 'the baseline is roughly fine' 'just get numbers out first' 'this many seeds is enough'. Violating any one item = serious dereliction of duty.

  1. Never let a watered-down comparison slide (unfair baseline):baseline mustsame data same split + equal tuning budget (each tuned to its best) + take the currently strongest available implementation. Using default hyperparameters / less tuning / a weakened baseline = watering down, the improvement is an illusion → fair_baseline critical. Dacrema 2019 (1907.06902): 18 algorithms, 6 beaten by simple heuristics——a comparison of optimized vs unoptimized simply cannot prove advancing SOTA。
  2. Never let a hypothesis be unfalsifiable:every hypothesis must have adisproof condition(what result could overturn it, quantifiable). Something un-overturnable like 'my method is better' = packaging, not science → falsifiable critical. Falsifiable ≠ falsified——the design mustgive the hypothesis a chance to be overturned。
  3. Never use single-run lucky numbers, and never pass off seeds/folds as independent samples to fake power:multiple runs estimate algorithmic randomness, but the n for formal power must be the independent units allowed by the design.power_check with d=0.5, 64 per group only answerstwo-independent-sample means comparison; paired CV, within-patient repeats, cluster/mixed designs must use matched methods or simulation. When the effect size has no source, give sensitivity/MDE, don't report a falsely precise single-point n.
  4. Never omit random seeds (especially cuDNN deterministic / PYTHONHASHSEED / DataLoader worker seeds):only set torch.manual_seed ≠ Reproducible.PYTHONHASHSEED Must be set before process startup; cuDNN must deterministic=True+benchmark=False; multi-process data loading must fix the worker seed——these are most commonly missed; miss one and the next run will drift. Reproducible > elegant.
  5. Never declare a component contributes without ablation and isolation: every innovative component must have anablation that removes it individuallyto prove its contribution; making causal/contribution claims ("proving X contributes") must be paired with anegative control(e.g., random labels) + equal tuning budget to rule out alternative explanations → otherwise ablation_isolation warn (unclean attribution = a point reviewers will question).
  6. Never hype heuristic lint as "proving the control is absolutely fair / the hypothesis is absolutely falsifiable":plan_lint/plan_gate isheuristic + reads your claims, what it checks is "whether it exists/is complete",it does not judge for you whether the baseline was actually tuned enough or whether the falsification conditions are set reasonably(GIGO). Fairness/falsifiability'sfinal judgment still requires human/domain judgment; the rigor score is a count-based deduction system (an auditable relative starting point), not ground truth.Honestly mark the boundaries; do not pretend to have checked everything.
  7. Never fit on the full dataset before splitting, and never flow a derived evaluation set back into training: prevent leakage via data-engineering (safe_split/split_leakage); derived sets (noise-added/missing/cross-domain)only touch features, not labels, fix the seed, and are for evaluation only——this is derive_eval_set 's iron rule; do not break it in your plan.
  8. Never overwrite the original primary endpoint or analysis chain after seeing the data: keep frozen nodes, the plan hash, and the original authorization; a new endpoint can only be marked EXPLORATORY_ENDPOINT, stating the triggering evidence, time, and user authorization. Comparative studies must also explicitly state whether the data, compute, tuning, and evaluation protocols are aligned.

Self-check trigger words: when you want to say "the baseline with default config is fine / this hypothesis definitely holds / let's just run it once and see / 5 seeds are enough / the seed is set to torch and that's enough / the ablation can be added next time / lint is green so it's fair"——stop, you have probably violated NEVER items 1/2/3/4/5/6, or missed the ASK rework/fairness/power decision.

Instruction flow: when to call which script (the engine is already in place; personally selftest until exit 0, call it directly, do not rewrite)

5 scripts in scripts/;plan_gate/plan_lint imports _shared(canonical bootstrap),power_check/target_chain pure stdlib (statsmodels optional; if missing, degrade to normal approximation and mark [APPROX]);research_package_gate reuses plan_lint as the final delivery gate. Before running on Windows set PYTHONUTF8=1。

① Control fairness/falsifiability critical gate → critical fail exit 1 (the soul of this skill)

# 编排 plan_lint + power_check + 显式声明 → light.findings.v1（fair_baseline/falsifiable critical）：
python scripts/plan_gate.py --spec plan_spec.json --report plan_findings.json   # 对照放水/不可证伪 → exit 1
# 交总控聚合（stage 5 确认点，critical fail → exit 1 确定性阻断）：
python ../light-orchestrator/scripts/run_checkpoint.py --file .light/passport.yaml --stage 5 \
    --findings plan_findings.json --write --ts 2026-06-19T11:00
# research-plan 自身门 fail = 在 stage 5 内改方案修复（无 ROUTES[5] 出边）；reroute 对 stage-5 给 manual 是正确兜底。

plan_spec.json:{project, matrix(或 matrix_file), hypotheses[{id,statement,falsifier}], baselines[{name,fairness}], power{effect_size,n_seeds,n_comparisons,correction}}(format see examples/plan_spec.example.json). Among these, n_seeds is a legacy field compatible with the existing gate,it can be filled only when the seed-level run itself is the independent unit of the estimand; otherwise omit it, let the gate honestly skip, and write the power/sensitivity evidence for the matched design into the plan and preregistration.

② Target chain freeze gate: the primary endpoint, counterevidence, and actions cannot be broken in the chain

python scripts/target_chain.py --input templates/target-chain.example.json

The in-repo template deliberately does not prefill research facts; running it directly should exit 1. It can pass only after the acyclic target chain, risk ledger, comparison fairness statement, and user authorization are filled in; PASS It does not prove that an effect exists, that the sample is sufficient, or that the method is effective; it only proves that the plan structure, timing, changes, and authorization fields are closed.

③ Failure tree/guardrail gate: not only write success criteria, but also write failure, inconclusive, and kill criterion

python scripts/failure_tree_gate.py --input failure-tree.json --as-of 2026-07-05 \
  --json-out .light/failure_tree_report.json

failure-tree.json Can start from templates/failure-tree.example.json . Each hypothesis must have success/failure/inconclusive three branches, and each branch must provide a quantifiable condition, an enumerated action_kind and claim_impact; when guardrail/counter-metric is missing, a reason for non-applicability must be given.FAIL must not be handed off to experiment-coding;WARN must be stated in the final package's warning_decisions as lowering the claim/adding experiments/user authorization.

④ Experiment matrix four-elements self-check + statistical power back-calculation (orchestrated by plan_gate, can also be run separately)

python scripts/plan_lint.py --file experiments/experiment_matrix.md   # 四要素缺项 exit 1 + 语义弱校验 + 严谨性评分
python scripts/power_check.py --effect 0.5 --n 5                       # 仅当 5 是每组独立观察；实际 power≈0.11
python scripts/power_check.py --effect 0.5 --target-power 0.8          # 双样本 t 反推每组 64，不泛化到复杂设计
python scripts/power_check.py --effect 0.5 --n-comparisons 10 --correction bh   # 多重比较校正后反推（更大 n）

If the unit of analysis is not an independent group observation,disable this closed-form result, and in the plan write method=simulation/MDE and the data-generating assumptions; see resource map Step 3 for details.

⑤ Complete plan package delivery gate: complete artifacts, complete reports, complete warning decisions, complete user authorization

# 最终交 experiment-coding 前必须跑 --final；WARN 只有在 warning_decisions 记录处理/降 claim/用户授权时才允许交接。
python scripts/research_package_gate.py --manifest plan_package.json --final --json-out research_package_report.json

plan_package.json Start with templates/plan_package.manifest.example.json . The gate will check PROJECT_PLAN.md、experiment_matrix.md、target_chain_report.json、failure_tree_report.json、plan_findings.json, preregistration package, reproducibility checklist, warning decisions, and handoff user authorization. Itdoes not replace domain/statistical/ethics review；PASS only indicates that the plan package evidence chain is closed,WARN indicates there are authorized claim-lowering/follow-up handling items.

⑥ Rework destinations: what others rework to research-plan (7→5 / 13→5)

# 下游产"结果不支撑假设"findings（result-analysis）→ 总控 reroute 建议回边 7→5（带"哪条假设没撑住"）：
python ../light-orchestrator/scripts/reroute.py --findings result_findings.json --stage 7 --passport .light/passport.yaml
# 用户拍板回炉后落一等回边（记在 stage5，不破坏拓扑）→ research-plan 重规划：
python ../light-orchestrator/scripts/passport.py add-back-edge --to 5 --from 7 \
    --root-cause "result-analysis 判 H1 未被结果支撑" --evidence-ptr "<reroute 给的指针>"
python ../light-orchestrator/scripts/passport.py validate --file .light/passport.yaml   # 回边不破拓扑 → 仍 PASS

Each script's --selftest/--help is the interface; for usage and known pitfalls see references.md(DVC/MLflow/W&B/Hydra/Sacred/ statistical power/preregistration/reproducibility protocol, verified tool by tool from primary sources).

Academician-level deep dive: the five are the passing line (blueprint §4.3-5, not bonus points)

① Experiment matrix four elements complete (not a macro narrative, but an executable matrix)

One runnable experiment per row,hypothesis→variables→metrics→stopping conditioncomplete (EXP-Bench 2505.24785: "design" and "conclusion" are the easiest to go astray).plan_lint Turn missing hypotheses/missing stopping conditions/decision disconnected from metrics from blind spots into line-by-line prompts. Stopping conditionsmust be quantifiable(borrowing from AI-Scientist v2's explicit stopping conditions at each stage: convergence + ≥2 datasets / budget exhausted); purely qualitative "good results" cannot be accepted → warn.

② Fair comparison + isolated ablation + statistical significance (critical + warn)

  • Fair comparison (critical): baseline with equal tuning budget, same data and same split, take the strongest available (Dacrema 2019). Giving slack = illusion of improvement = critical.
  • Isolated ablation (warn): remove innovative components one by one to prove the source of contribution, and pair causal claims with negative controls to rule out alternative explanations. All at once = unclean attribution.
  • Statistical power (warn): perform power/sensitivity according to the design's independent unit, report error bars/CI, correct for multiple comparisons; multiple seeds are used to estimate algorithmic randomness, and cannot automatically substitute for the sample size of patients/clusters. A single-run lucky number does not count as evidence.

③ Falsifiability (the design must let the hypothesis be overturned)

Each hypothesis hasa disproof condition(what result can overturn it). An experiment with no falsifiability = not science, but packaging (Popper) → critical.Falsifiable ≠ falsified。

④ Reproducible full traceability (seeds/environment/versions/splits)

NeurIPS checklist: code + environment versions + weights + hyperparameters + multi-seed error bars.v2 emphasizes the seeds most often missed:PYTHONHASHSEED(before process start), cuDNN deterministic=True+benchmark=False, DataLoader worker seeds. Fixing seeds (reproducible) ≠ multi-seed (repeated experiments); both are needed, don't mix them up.

⑤ Academician will follow up (rehearsal)

Can your experimentfalsify your hypothesis? Is the comparisonfair(has the baseline been tuned enough)? Can the ablationcleanly isolateeach component's contribution? Does the effect's CI/precision support the claim? Are power/sensitivity's independent units aligned with the design? Does it still hold with a different seed? — If you can't answer, the plan hasn't reached the passing line.

Closing self-check (go through before issuing verdict / before writing back to master control)

  • Does every row of the experiment matrix haveall four elements(hypothesis/variables/metrics/stopping condition)? Is the stopping conditionquantifiable? (plan_lint run, missing items cleared)
  • Is question→population/unit→estimand→outcome/analysis aligned? Are the statistical unit, randomization unit, and analysis unit not swapped?
  • Are the primary endpoint's measurement instrument, operational definition, unit, minimally meaningful effect, and missingness policy locked?
  • Does the analysis family state the independence assumption? If the statistical/randomization/analysis units are inconsistent, is there a reasonable rationale?
  • Does every baseline declarefairness conditions(equal tuning/data/same strongest available)? Are any given a pass? (fair_baseline not unfair)
  • Does every hypothesis havea quantifiable falsification condition? Is there any unfalsifiable hypothesis? (falsifiable not critical)
  • Does every hypothesis clearly state the success/failure/inconclusive three branches, actions, and claim impact? Do guardrail/kill criterion have thresholds and trigger actions? (failure_tree_gate)
  • Is every strong innovation claim traced from innovation_engine 's anti_collage to a discriminating experiment that distinguishes the new mechanism vs the old explanation? If not, is it downgraded to an engineering increment/systematization?
  • Does every innovation hypothesis havean ablationisolating its contribution? Are causal claims accompanied bynegative controls? (ablation_isolation)
  • Do SESOI/effect sizes have a source and range? Does the power method match the design? You didn't treat seed/fold/repeated measures on the same subject as independent n, right? Do complex designs have simulation/MDE?
  • Are primary/secondary/exploratory, comparison family, exclusion, stopping/guardrail locked? Is the preregistration version/hash/status honest?
  • Does the reproducibility checklist havefull seed coverage(cuDNN/PYTHONHASHSEED/worker)? Are environment/version/splits accounted for?
  • Is the derived evaluation set spec correct (only change features, don't touch labels/ fixed seed/ evaluation only) and sent back to data-engineering for construction?
  • research_package_gate.py --manifest plan_package.json --final Has it been run? If verdict=WARN,warning_decisions is it clearly stated whether to fix/downgrade the claim/get user authorization? If FAIL, is it still fixing the plan at stage 5 instead of handing off downstream?
  • You haven't **blown heuristic lint up into "proving the comparison is absolutely fair/the hypothesis is absolutely falsifiable"**, right? (honestly mark boundaries; final judgment requires a human)
  • Is it coming back from rework (7→5/13→5)? Bring "which hypothesis didn't hold up/reviewer questions"replan、stop and ask the user?

Name-reality alignment (honest, not hyped as a selling point)

True increment (delivered in v2, self-tested):① Comparison fairness/falsifiability critical gate producer(plan_gate.py,v2 net-new wiring)—— Orchestration comes from plan_lint(experiment matrix linter) + power_check(statistical power) + plan explicitly declared → produces light.findings.v1(producer= research-plan):Comparison watered down / hypotheses lack falsification conditions → critical(aligned with STAGE_GATES[5]=[fair_baseline,falsifiable]), ablation not isolated / underpowered → warn, aggregated by run_checkpoint --stage 5 exit 1. v1's plan_lint/power_check is a pure linter/tool, zero output light.findings.v1(grep-verified), findings wiring is v2-new. ② Changed plan_lint 's hardcoded parents[2]/_shared to canonical bootstrap (would necessarily break after the v2 repo root moved up one level). ③ Reproducibility checklist added the most commonly missed seeds(cuDNN deterministic/PYTHONHASHSEED/worker seeds, missed in v1). ④ Experiment matrix template adds comparison fairness declaration + falsification conditions + negative control column(docked to plan_gate's three critical/warn gates). ⑤ Round 2 adds question/estimand→design→power/sensitivity→preregistration provenance→checkpoint→downstream/7→5 eight-step resource closed loop and registration template; ⑥ adds target_chain.py, turning the frozen target chain, plan hash, user authorization, post-data exploratory downgrade, acyclicity check, risk ledger and comparison fairness fields into executable gates; Round 3 further adds statistical/randomization/unit of analysis, endpoint measurement operationalization, minimum meaningful effect, missing-data strategy and independence assumptions, blocking the common failure modes of "beautiful plan wording but untestable endpoint / seed treated as independent n / cluster swapping the unit"; does not expand the overall critical surface. ⑦ adds research_package_gate.py, turning "plan text + matrix + target-chain report + plan_findings + preregistration + reproducibility checklist + warning decisions + user authorization" into a final delivery gate;PASS/WARN before experiment-coding can be handed off,WARN must have claim downgrade/follow-up handling and user authorization,FAIL must not go downstream. ⑧ Round 3 adds failure_tree_gate.py: turning success/failure/inconclusive branches, guardrail/counter-metric, kill criterion, default action on resource exhaustion and post-data amendment policy into machine-readable gates, and wires into research_package_gate; a final package without a failure tree or with a failure-tree report that is not PASS/WARN must not be handed downstream,WARN must write warning_decisions.

What a bare model already knows how to do (no hype):"baseline must be fair" "do ablations" "run more seeds" "hypotheses must be falsifiable"——bare Opus would say all of that. This skill's value = ① Turning comparison fairness/falsifiability into deterministic critical machine-readable gates + deterministic blocking(a bare model says fair with its mouth but still lets watered-down plans through with its hands; the orchestrator can't read that); ② Power/sensitivity up front + multiple-comparison linkage, and explicitly refusing to treat seed/fold as independent n; ③ Machine-readable findings + root-cause rework (research-plan is the 7→5/13→5 rework destination; a bare model has no such orchestration closed loop).

Honest laggards (known not done):

  1. plan_lint is a heuristic linter with boundaries: checks the four elements are complete / judgments are quantifiable / judgment-metric semantic alignment (offline semantic_sim is weak across languages)/ ablation coverage / negative control / multiple-comparison family count——Never "proves the control is absolutely fair / the hypothesis is absolutely falsifiable"; the rigor score uses a count-based deduction system,not truth values, not an ARA semantic cognition review. The final fairness/falsifiability judgment still requires human/domain judgment.
  2. fair_baseline relies on declarations + signals; it does not judge for you whether the baseline was tuned enough: the gate reads the plan'sexplicitly declared fairness conditions; "whether equal tuning budgets are truly equal" must be checked by a human; if undeclared → warn prompting to add it,does not fabricate "favorable treatment"(GIGO)。
  3. falsifiable checks "whether there are disproof conditions", not "whether the disproof conditions are correct": whether the disproof conditions are reasonable and truly attainable requires domain judgment.Falsifiable ≠ falsified。
  4. narrow power coverage and extremely sensitive to assumptions:power_check only applies to the two-independent-sample means approximation; the d source must be SESOI/external evidence/a shrunk pilot. paired/ANOVA/proportions/correlation/cluster/mixed/repeated-CV use the corresponding Power class or simulation; fixed N report MDE/sensitivity. statsmodels missing degrades to normal approximation marked [APPROX]. The machine will not verify the authenticity of the effect source.
  5. division of labor with data-engineering power (no overlap):data-engineering sample_size_check = data scale before proposing an ideaempirical rough screening(per-class minimum sample/EPV, no effect size, gates idea 2⊣3); research-plan does formal power or sensitivity after the estimand/design is clear. power_check is only the two-sample t subset of it, and does not mean all plans have had formal power.
  6. do not reinvent experiment tracking/versioning/sweep wheels: DVC/MLflow/W&B/Hydra/Sacred are mature tools; this skillteaches selection + gives real commands(references), does not build in a tracking server/data version repository; tiered selection (light/standard/full), do not force a heavy one onto a small project (Snakemake has poor Windows compatibility → WSL/invoke/make).

Standard output artifacts:PROJECT_PLAN.md(research plan, handed to experiment-coding) · experiments/experiment_matrix.md(experiment matrix) · preregistration.md(frozen plan + registry provenance) · reproducibility-checklist.md(reproduction checklist) · plan_findings.json(control fairness/falsifiability gate) · target-chain.json(frozen goal chain + authorization/change ledger) · failure-tree.json + failure_tree_report.json(success/failure/no conclusion/guardrail/kill criterion) · derive_spec(derived evaluation set back to data-engineering) · plan_package.json + research_package_report.json(final delivery gate evidence). Save to .light/; passport registration handed to memory-pm, plan changes written back to .light/decision_log。

References (three-level progressive disclosure: read when needed)