scaffold-agent-tests

Skill: Generate Agent Test Suite

Purpose

Read an LLM Agent's implementation and its test contract (specs/agent-test-modeling.md instance), and generate a test suite traceable to the contract—precise assertions for deterministic parts, oracle tests for non-deterministic parts, golden dataset, and evaluator harness.

Core Objective

Primary objective: produce a set of agent test code that follows rules/standards-agent-testing.md , where every assertion is traceable to a clause of the contract.

Success criteria(all must be satisfied):

✅ Contract located: read agent implementation + agent-test-contract document
✅ Test matrix generated: capability boundaries / input contract / tool boundaries / write-back preconditions each mapped item by item to tests
✅ Oracle selection correct: use precise assertions for deterministic behavior; use contract / trajectory / rubric / golden oracle for non-deterministic behavior
✅ Golden dataset persisted: at least 1 each of positive + boundary + exception, including Covers traceability anchor
✅ Real-model tests isolated: tests that hit real models are marked with eval marker, unit pipelines use mock LLM
✅ Traceable: each test's Covers points to a contract ID or upstream AC

Acceptance test: can a developer, without reading the agent source code, understand what each test guards based solely on the generated tests + contract?

Scope Boundaries

This skill is responsible for：

Reading agent implementation + test contract, generating test matrix
Generating mock LLM unit tests, golden dataset, evaluator harness
Selecting oracles for non-deterministic behavior, writing precise assertions for deterministic behavior
Attaching Covers traceability anchor to each test

This skill is not responsible for：

Running tests → use automate-tests
Reviewing test quality / coverage → use review-testing
Writing test contract documents → the contract is written by humans according to specs/agent-test-modeling.md ; this skill consumes contracts, it does not create them
Fixing failing tests / debugging agents → use orchestrate-repair-loop

Handoff point: test generation complete → hand off to automate-tests to run, review-testing to review.

Prerequisites

Agent implementation source code is readable
A test contract that follows specs/agent-test-modeling.md exists;when missingfirst prompt the user to write a contract according to that spec, do not invent capability boundaries on your own

Execution Flow

1. Locate contract and implementation

Read agent-test-contract document (frontmatter agent_ref points to implementation)
Contract missing → terminate and prompt: first follow specs/agent-test-modeling.md to write a contract

2. Generate test matrix

Map each section of the contract to test entries:

Contract section	Test type	oracle
Capability boundaries	Positive + negative behavior tests	Trajectory / contract
Input contract (missing field identification)	Boundary + exception tests	Trace (assertion follow-up, no write-back)
Tool call boundaries	Trace test	Trace (forbidden set not present)
Write-back preconditions	Rejection test when preconditions are not met	Trace
Golden Cases	Regression test set	List by contract decision method

3. Choose oracle and write tests

Deterministic logic(schema validation, tool parameters, permissions) → exact assertions (follow standards-test-code AAA + three naming elements)
Non-deterministic output → contract / trace / rubric / golden oracle (follow standards-agent-testing §2）
unit pipeline uses mock LLM / recorded replay; real model tests are tagged eval marker

4. Create golden dataset

Generate versioned golden data files from the contract Golden Cases table
Each entry contains input / expectation / decision method / Covers

5. Generate evaluator harness

For rubric / semantic / statistical decision types, generate evaluators (LLM-as-judge or semantic matcher)
Output golden set pass rate, compare with contract pass_threshold

6. Summary

List generated files, each test's Covers, unit vs eval layering
Follow-up hints: automate-tests run, review-testing review

Limitations

Hard Boundaries

Do not fabricate agent capability boundaries when the contract is missing (ask for the contract first)
Do not mock the logic of the agent under test itself (only mock model non-determinism)
Do not weaken assertions or delete golden cases in order to pass eval
Do not run tests, do not modify the agent implementation (only generate test artifacts)

Skill boundaries (avoid overlap)

Run tests → automate-tests
Review test quality → review-testing
Debug / fix → orchestrate-repair-loop

Self-Check

Read the agent implementation + test contract
Test matrix covers all contract sections (capability boundaries / input / tools / write-back / golden)
Use exact assertions for deterministic cases, oracle for non-deterministic cases
Golden set contains positive + boundary + exception cases, each with Covers
Real model tests are tagged eval marker, unit uses mock LLM
Every test can be traced back to the contract or upstream AC

Examples

Example: requirements clarification agent

User: "Generate tests for the clarification agent."

Agent:

Read agent-test-clarification.md(contract) + src/agents/clarification.py
Generate matrix: missing field identification (trace), complete write-back (contract), empty input (trace), forbidden tools (trace)
Write tests:
- unit: mock LLM returns recorded response, assert trigger ask_user does not trigger write_requirement
- eval (tagged marker): real model runs golden set, rubric evaluates clarity of follow-up wording
Create golden/clarification.jsonl(3 entries: missing acceptance / complete information / empty input)
Generate evaluator: calculate golden pass rate, compare with pass_threshold: 0.9
Summary: 4 unit + 1 eval suite, each attached to Covers: ACME-REQ-08#AC1/AC3; hint automate-tests to run