Positioning
Behavior evaluation of high-frequency process skillsbehavior evaluation(dynamic): run N times each with/without skill → pass-rate comparison →
failure cases feed back into the skill body. Complements skill-description-audit(static description compliance).
When to use
- After a skill change (did behavior actually improve)
- Quarterly evaluation (baseline cadence)
- New skill acceptance (establish baseline first after delivery)
Input
- Target skill (first batch: code-review / project-handoff / writing-plans / stage-gate)
- Evaluation record directory:
.scratch/neonforge-v1/skill-eval/<skill>-<YYYY-MM-DD>/
Process
- [ ] 1. 定义代表任务：3-5 个（见下节）
- [ ] 2. 无 skill 基线：每任务跑 N 次（N≥3，fresh agent、无对话种子）
- [ ] 3. 有 skill 组：同任务同 prompt 跑 N 次（fresh agent + 加载目标 skill 指令）
- [ ] 4. 逐次判定 pass/fail（按任务 AC，机器可验证优先——文件存在/输出匹配/命令结果）
- [ ] 5. 汇总 pass-rate 对比表（任务 | 无 skill | 有 skill | Δ）
- [ ] 6. 失败案例分析：成功/失败样例各取代表，定位 skill 改进点
- [ ] 7. 产出报告 + 改进项写回 skill SKILL.md（或作为下轮验收项）
- [ ] 8. 结果入 SKILLS-MAP 或该 skill README（基线表）
Representative task definition (3-5)
Each task = oneindependent prompt + decidable AC:
| Requirement | Description |
|---|---|
| Covers core capability | The task must hit the skill's signature capability (e.g., stage-gate's "run S{N} gate") |
| Has representative failure modes | When selecting tasks, predict "how it would go wrong without skill" (missed verification/prose assertions/verify-only-no-fix) |
| AC machine-verifiable | Preferred: file exists and contains X / command output matches / assertion count correct; secondary: structured report fields complete |
| Realistic scale | Use the minimal scenario of a real repository (same-repo fixture), not toy examples |
Run discipline
- fresh agent: each run in each group is a new session (no conversation seed) — prevents memory contamination; the with-skill group only additionally loads the target skill instructions, all other prompts are verbatim identical.
- Same prompt verbatim identical: the only difference between the two groups is "whether the skill is loaded".
- Judge independent of executor: pass/fail is decided by AC, not by "agent self-reported success" (verification-before-completion).
- Record: each run's task prompt, AC, artifact paths, judgment — written to disk in the evaluation directory, report reviewable.
Report (pass-rate comparison table)
# Skill Eval: {skill}（{date}）
| 任务 | 无 skill (N) | 有 skill (N) | Δ | 结论 |
|------|-------------|-------------|---|------|
| 任务 1 | 1/3 | 3/3 | +2 | skill 有效 |
| 任务 2 | 2/3 | 2/3 | 0 | 无效/无差异——改进点：… |
## 失败案例分析（反哺）
- 有 skill 仍失败 ×2 → 改进项：{具体到 SKILL.md 哪节缺什么}
## 改进项（低分 skill 必出）
1. …（具体、可验收）
Acceptance
- Each task has N≥3 runs with/without skill (total ≥6 runs/task), records written to disk
- Output a pass-rate comparison table (with Δ and conclusion)
- Low-scoring skills (with skill shows no significant improvement or still fails) get concrete improvement items
- Improvement items have been fed back (written back to SKILL.md or listed for the next acceptance round)
- Baseline results entered into SKILLS-MAP / skill README
Boundary (division of labor)
| Adjacent skill | Division of labor |
|---|---|
skill-description-audit |
Static description compliance (description↔body); skill-eval = behavior evaluation (dynamic pass-rate) — compliance first, then behavior |
verification-before-completion |
Source of judgment discipline (AC evidence judgment, don't trust self-reports) |
delegated-research |
Not applicable — evaluation is a controlled experiment, not research |
Anti-patterns
- N<3 or inconsistent conditions between the two groups (conclusion has no statistical significance)
- Running both groups in the same session (skill instructions leak into the baseline group)
- Judging by "feeling" instead of AC (treating self-reported success as pass)
- Only reporting the comparison without feeding back (skill didn't improve after evaluation)
- Tasks don't cover the signature capability (evaluated but didn't evaluate the right thing)