add-eval

Add Eval

Add or update an LLM eval test for a specific tool or collection. This skill creates a new eval scenario or updates an existing one to cover new tools.

Use this skill when:

You added a new tool with /add-tool and want eval coverage
You modified a tool and need to update the eval scenario
You want to add a new workflow scenario to an existing collection
/build-evals skipped the collection because eval files already exist

Prerequisites

Before running, ensure:

The collection exists (src/umbraco-api/tools/{collection}/index.ts)
The project builds: npm run build
An LLM API key is available (eval tests use Claude Agent SDK — works with a Claude Code subscription or ANTHROPIC_API_KEY)

Arguments

Collection name (required): the collection to add eval coverage for (e.g. /add-eval form)
Tool or scenario hint (optional): what to test (e.g. /add-eval form "copy form workflow")

If no hint is provided, compare the tools in the collection against existing eval test coverage and suggest what's missing.

Agents

Agent	When to use
`eval-test-creator`	Creating or updating eval test files (Step 3 or 4)

Critical Rules

BUILD BEFORE RUNNING. Eval tests run against dist/index.js. Always npm run build first.

RUN COMMANDS SEPARATELY. Always run build and test as separate Bash calls. Never chain with &&.

ITERATE ON PROMPTS. Eval tests are probabilistic. If a test fails, the fix is usually in the prompt.

VERBOSE DURING DEVELOPMENT. Always set verbose: true when creating or debugging.

Workflow

Step 0: Check Eval Infrastructure

Check if the eval setup exists:

tests/evals/helpers/e2e-setup.ts
tests/evals/jest.config.ts

If the setup doesn't exist, tell the user to run /build-evals {collection} first to create the infrastructure. This skill does not create eval setup files.

Step 1: Understand Current Coverage

Read the collection's tools and existing eval tests:

src/umbraco-api/tools/{collection}/index.ts — all tools in the collection
tests/evals/{collection}-*.test.ts — existing eval scenarios
Each eval test file — to understand which tools are covered and in what scenarios

Build an inventory:

Tools with eval coverage (appear in a scenario's tools and requiredTools arrays)
Tools without eval coverage
Existing workflow scenarios

Step 2: Decide: Update or Create

Based on the inventory and the user's request:

Update an existing scenario when:

A new tool extends an existing workflow (e.g. adding copy-form to a CRUD scenario)
The new tool is a natural addition to an existing test's prompt

Create a new scenario when:

The new tool represents a different capability (e.g. search, hierarchy, import/export)
Adding it to an existing scenario would make the prompt too complex
No existing scenario covers the tool's domain

Step 3: Update Existing Scenario (if applicable)

If updating an existing eval test:

Add the new tool to the COLLECTION_TOOLS array
Add the tool to requiredTools if it must be called
Update the prompt to include steps that exercise the new tool
Keep the prompt clear and sequential — don't make it overly long

Build and run:

npm run build

npm run test:evals -- --testPathPatterns="{collection}"

If the test fails, iterate on the prompt. Common fixes:

Make instructions more explicit
Add a search/list step to find IDs dynamically
Simplify the workflow if the prompt is too complex

Step 4: Create New Scenario (if applicable)

Use the eval-test-creator agent to create tests/evals/{collection}-{workflow}.test.ts. The agent owns the test template and patterns.