Improvement method = do SGD on the idea. Objective function || Experiment(idea) − Target result ||，
"Analyzing why an experiment doesn't work" is exactly computing the gradient.。
From this we get the only efficiency criterion:whether each loop produced a verified essential technical reason.
No → the gradient of this loop is random, and the next change is equivalent to rolling dice.
A failed experiment is not a waste—it rules out a path, provided that path waswritten down.
(fell into experiments/LOG.md, see research-project）。
1. Designing Experiments
Core principle: reduce exploration points
The more unknowns in an experiment, the less attributable the result. Two kinds of decomposition, usually used at the same time:
| Decompose the pipeline | Decompose the experimental setup |
|---|---|
| Break the idea into components, start from thecontrollablepart, and gradually add exploratory modules | Start from asimple setting , gradually increase the difficulty, and finally move to the real setting |
Simple experiment: only the core technical challenge exists
明确解法要解决的 core technical challenge
→ 找出当前数据里还有哪些额外的 technical challenge
→ 寻找或构建不含这些额外 challenge 的数据
→ 在这样的数据上验证解法
The advantage is low cost, andno interference from other challenges, so it can cleanly determine whether the solution itself works. Order: verify correctness on simple data → improve on simple data → tune work on real data.
The minimal viable three steps for exploratory experiments
Step 1: Define a binary feasibility criterion. What does this experiment aim to verify? What is the success criterion?It must be binary, clear, and measurable(yes/no). If you cannot write a binary criterion = you have not yet thought clearly about what to verify, and doing the experiment at this point is a waste.
Step 2: Design the smallest experiment. Use this sentence to force it:
"If I had only one day, what is the simplest verification I could do?"
Step 3: Judge against the criterion, leading only to one of three directions.
| Result | Next step |
|---|---|
| Success | Design a more rigorous and larger-scale experiment to consolidate; start increasing complexity |
| Failure | Ruled out one path at minimal cost. Analyze whether the idea is wrong or the implementation is wrong, and pivot or terminate accordingly |
| Ambiguous | **The experiment was not designed well enough, and the criterion was not defined clearly.** Go back to Step 1 and redefine |
The third row is most often handled incorrectly: when getting ambiguous results,do not "run a few more settings". Repeating an ambiguous result several times is still ambiguous.
2. Diagnosing Why an Experiment Doesn't Work
表面原因 → 本质技术原因 → 用实验验证这个原因
All three steps must be completed.Only after confirming that it really is a "technical reason" does changing the algorithm make sense.
Step 1 · Surface cause: which variable, when changed, makes it worse (changed data / added a module / changed parameters). The method is tocompare the two groups that work and do not work, and list their specific differences.
Step 2 · Essential cause: ask "why" about the surface cause, andlist as many candidates as possible. Two categories:
- The code has a bug
- The "xx algorithm" really does have a problem on the "xx data": Hyperparameters not set right / missing a few tricks / the algorithm itself doesn't work
Ways to distinguish the 2nd category:Look at why related papers work and what tricks they use
（literature-review Search → pdf-explore read implementation details),
Run ablations on the algorithms of related papers See which part is key.
Some ideas don't work on their own, and only work with tricks added (NeRF + positional encoding).
Step 3 · Validation: design experiments to falsify one by one.Skipping validation and directly modifying the algorithm = guessing the gradient.
Output template
When the user says "the experiment doesn't work",don't directly give improvement suggestions. First produce this table; if a cell is missing, go fill it in; if you can't fill it in, say explicitly "this cell requires you to provide X".
## 诊断：<一句话现象>
**表面原因**：<哪个变量一改就变差>
**对照**：work = <配置> 不 work = <配置> **差异**：<具体列出，不写"设置不同">
| # | 候选本质原因 | 支持证据 | 反对证据 | 怎么用一个实验证伪 |
|---|---|---|---|---|
| 1 | 代码 bug：<具体哪里> | | | |
| 2 | 超参：<哪个> | | | |
| 3 | 缺 tricks：<论文 X 用了 Y> | | | |
| 4 | 算法本身不行：<为什么> | | | |
**下一个实验**（只验证一条，exploration points = 1）
- 验证第 <N> 条 - 二元标准：<若…则该原因成立> - 最小设计：<一天能做完的版本>
**还需要你提供**：<看不到的信息，如训练日志、baseline 配置>
Self-check:Is the "next experiment" derived from the table, or did you think of it first and then fill the table backwards? The latter means you're rationalizing intuition, not seeking the gradient.
A filled-in example:references/diagnosis-example.md (a complete diagnosis of the drop in performance after depth prior supervision). It demonstrates three things: the "difference" cell must be written down to the specific configuration, the candidate causes must include a "code bug" category, and the candidateexecution order should be ordered by troubleshooting cost rather than by likelihood——do the ones that can be ruled out in ten minutes first.
Three, what experiments the paper needs
Experiment planning is back-derived from the three questions the paper needs to answer:
| Question | Experiment |
|---|---|
| How to prove the method is stronger than existing methods? | comparison: compare with related,newer baselines. When the task is too new and there is no baseline, construct variants of your own method |
| How to prove each module is effective? | ablation: ① one large table + visualization, listing the impact of core contributions and important components; ② several small tables, each covering the design choices of one module (sensitivity to hyperparameters, sensitivity to input quality, impact of removing a certain choice) |
| How to show the upper limit of the method? | Do a demo on more challenging data |
The third item determines impact, and the criterion is:
Do a demo for the downstream research community—make it for the people who will cite this paper, not for yourself or the general public.
After the algorithm works on the target data,try to run it on newer, harder data and push the algorithm to its limit. "Everyone wants to see how far an algorithm can go" is itself a great contribution. Counterfactual: if Neural Body had not done the street dance data, it would not have gotten a best paper candidate.
Missing ablation / missing important baseline / missing important metric / data too simple—these four are all common rejection reasons,
see paper-self-review。
Connection
Write the diagnosis conclusion and the next experiment back into experiments/LOG.md and PROJECT.md §4（research-project).
When the approach itself has a problem (rather than an implementation problem), go back to research-ideation。