A formal evaluation framework for Claude Code sessions, implementing the principles of Eval-Driven Development (EDD).
When to Activate
- Set up Eval-Driven Development (EDD) for AI-assisted workflows
- Define pass/fail criteria for Claude Code task completion
- Measure Agent reliability using pass@k metrics
- Create regression test suites for Prompt or Agent changes
- Benchmark Agent performance across model versions
Core Philosophy
Eval-Driven Development treats evals as the "unit tests of AI development":
- Before implementationBeforedefine expected behavior
- Continuously run evals during development
- Track regressions caused by each change
- Use pass@k metrics for reliability measurement
Eval Types
Capability Evals
Test whether Claude can complete tasks it could not complete before:
[CAPABILITY EVAL: feature-name]
任务：描述 Claude 应该完成的目标
成功标准：
- [ ] 标准 1
- [ ] 标准 2
- [ ] 标准 3
预期输出：对预期结果的描述
Regression Evals
Ensure changes do not break existing functionality:
[REGRESSION EVAL: feature-name]
基线（Baseline）：SHA 或检查点（checkpoint）名称
测试项：
- existing-test-1: 通过/失败（PASS/FAIL）
- existing-test-2: 通过/失败（PASS/FAIL）
- existing-test-3: 通过/失败（PASS/FAIL）
结果：X/Y 通过（之前为 Y/Y）
Grader Types
1. Code-Based Grader
Use code for deterministic checks:
# 检查文件是否包含预期模式
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
# 检查测试是否通过
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
# 检查构建是否成功
npm run build && echo "PASS" || echo "FAIL"
2. Model-Based Grader
Use Claude to evaluate open-ended output:
[MODEL GRADER PROMPT]
评测以下代码变更：
1. 它是否解决了所述问题？
2. 结构是否良好？
3. 是否处理了边缘情况？
4. 错误处理是否得当？
得分：1-5（1=差，5=优秀）
推理：[解释说明]
3. Human Grader
Flag for human review:
[HUMAN REVIEW REQUIRED]
变更内容：描述发生了什么变化
原因：为什么需要人工审查
风险等级：低/中/高（LOW/MEDIUM/HIGH）
Metrics
pass@k
"At least one success in k attempts"
- pass@1: first-attempt success rate
- pass@3: success rate within 3 attempts
- Typical target: pass@3 > 90%
pass^k
"All k trials succeed"
- Higher reliability requirements
- pass^3: 3 consecutive successes
- Used for critical paths
Eval Workflow
1. Define (Before Coding)
## 评测定义：feature-xyz
### 能力评测
1. 可以创建新用户账户
2. 可以验证电子邮件格式
3. 可以安全地对密码进行哈希处理
### 回归评测
1. 现有登录功能仍然正常
2. 会话管理保持不变
3. 注销流程完好无损
### 成功指标
- 对于能力评测：pass@3 > 90%
- 对于回归评测：pass^3 = 100%
2. Implement
Write code to pass the defined evals.
3. Run Evals
# 运行能力评测
[运行每项能力评测，记录通过/失败]
# 运行回归评测
npm test -- --testPathPattern="existing"
# 生成报告
4. Report
评测报告：feature-xyz
========================
能力评测：
create-user: 通过 (pass@1)
validate-email: 通过 (pass@2)
hash-password: 通过 (pass@1)
总体： 3/3 通过
回归评测：
login-flow: 通过
session-mgmt: 通过
logout-flow: 通过
总体： 3/3 通过
指标：
pass@1: 67% (2/3)
pass@3: 100% (3/3)
状态：准备好进行评审 (READY FOR REVIEW)
Integration Patterns
Pre-Implementation
/eval define feature-name
In .claude/evals/feature-name.md create an eval definition file
During Implementation
/eval check feature-name
Run current evals and report status
Post-Implementation
/eval report feature-name
Generate a complete eval report
Eval Storage
Store evals in the project:
.claude/
evals/
feature-xyz.md # 评测定义
feature-xyz.log # 评测运行历史
baseline.json # 回归基线
Best Practices
- Define evals BEFORE coding - Forces clear thinking about success criteria
- Run evals frequently - Catch regressions early
- Track pass@k over time - Monitor reliability trends
- Use code graders when possible - Deterministic beats probabilistic
- Human review for safety-critical - Never fully automate safety checks
- Keep evals fast - Slow evaluations are often not run
- Version-control evaluations together with code - Evaluations are first-class assets
Example: Add authentication (Authentication)
## EVAL: add-authentication
### 第 1 阶段：定义 (10 分钟)
能力评测：
- [ ] 用户可以使用电子邮件/密码注册
- [ ] 用户可以使用有效凭据登录
- [ ] 无效凭据被拒绝并显示正确错误
- [ ] 页面重新加载后会话依然持久化
- [ ] 注销后清除会话
回归评测：
- [ ] 公共路由仍可访问
- [ ] API 响应保持不变
- [ ] 数据库架构兼容
### 第 2 阶段：实现 (耗时视情况而定)
[编写代码]
### 第 3 阶段：执行评测
运行：/eval check add-authentication
### 第 4 阶段：报告
评测报告：add-authentication
==============================
能力：5/5 通过 (pass@3: 100%)
回归：3/3 通过 (pass^3: 100%)
状态：准予发布 (SHIP IT)