Build rigorous evaluations for AI agents using Anthropic's proven patterns.
Quick Reference
You MUST read the reference files for detailed guidance:
- Grader Types - Code-based, model-based, human graders
- Agent Type Patterns - Coding, conversational, research, computer use
- Roadmap - Steps 0-8 for building evals from scratch
- Frameworks - Harbor, Promptfoo, Braintrust, etc.
YAML Templates:
- coding-agent-eval.yaml - Coding agent template
- conversational-agent-eval.yaml - Support agent template
Annotated Examples:
- Example: Coding Agent - Auth bypass fix walkthrough
- Example: Conversational - Refund handling walkthrough
Core Definitions
| Term | Definition |
|---|---|
| Task | Single test with defined inputs and success criteria |
| Trial | One attempt at a task (run multiple for consistency) |
| Grader | Logic that scores agent performance; tasks can have multiple |
| Transcript | Complete record of a trial (outputs, tool calls, reasoning) |
| Outcome | Final state in environment (not just what agent said) |
| Evaluation harness | Infrastructure that runs evals end-to-end |
| Agent harness | System enabling model to act as agent (scaffold) |
| Evaluation suite | Collection of tasks measuring specific capabilities |
Grader Types (Quick Reference)
| Type | Methods | Best For |
|---|---|---|
| Code-based | String match, unit tests, static analysis, state checks | Fast, cheap, objective verification |
| Model-based | Rubric scoring, assertions, pairwise comparison | Nuanced, open-ended tasks |
| Human | SME review, A/B testing, spot-check sampling | Gold standard calibration |
See Grader Types for detailed comparison.
Capability vs Regression Evals
| Type | Question | Target Pass Rate |
|---|---|---|
| Capability | "What can this agent do well?" | Start low, hill-climb |
| Regression | "Does it still handle what it used to?" | Near 100% |
Capability evals with high pass rates "graduate" to regression suites.
Non-Determinism Metrics
| Metric | Measures | Use When |
|---|---|---|
| pass@k | At least 1 success in k attempts | One success matters (coding) |
| pass^k | All k attempts succeed | Consistency essential (customer-facing) |
Example: 75% per-trial success rate
- pass@3 ≈ 98% (likely to get at least one)
- pass^3 ≈ 42% (0.75³ all succeed)
Tracked Metrics
tracked_metrics:
- type: transcript
metrics: [n_turns, n_toolcalls, n_total_tokens]
- type: latency
metrics: [time_to_first_token, output_tokens_per_sec, time_to_last_token]
Attribution
Based on Demystifying evals for AI agents by Anthropic (January 2026).