evalview-agent-testing

Historical version and verification status

This is the complete historical guide merged on 2026-03-31 at 21:13:33 UTC, merge commit 0f40fd030c222f1c56ae9ef05d5115ac9fd140a0. The full SKILL.md and MIT license at that commit were checked on 2026-10-18 UTC and attribution is retained. This does not establish presence on the current branch, current installability, dependency compatibility, tool counts, costs, performance or rollback behavior. No source command was executed.

Historical merged original

Original metadata

name: evalview-agent-testing
description: Regression testing for AI agents using EvalView. Snapshot agent behavior, detect regressions in tool calls and output quality, and block broken agents before production.
origin: ECC
tools: Bash, Read, Write

EvalView Agent Testing

Automated regression testing for AI agents. EvalView snapshots your agent's behavior (tool calls, parameters, sequence, output), then diffs against the baseline after every change. When something breaks, you know immediately — before it ships.

When to Activate

• After modifying agent code, prompts, or tool definitions
• After a model update or provider change
• Before deploying an agent to production
• When setting up CI/CD for an agent project
• When an autonomous loop (OpenClaw, coding agents) needs a fitness function
• When agent output changes unexpectedly and you need to identify what shifted

Core Workflow

# 1. Set up
pip install "evalview>=0.5,<1"
evalview init              # Detect agent, create starter test suite

# 2. Baseline
evalview snapshot           # Save current behavior as golden baseline

# 3. Gate every change
evalview check              # Diff against baseline — catches regressions

# 4. Monitor in production
evalview monitor --slack-webhook https://hooks.slack.com/services/...

Understanding Check Results

Status	Meaning	Action
PASSED	Behavior matches baseline	Ship with confidence
TOOLS_CHANGED	Different tools called	Review the diff
OUTPUT_CHANGED	Same tools, output shifted	Review the diff
REGRESSION	Score dropped significantly	Fix before shipping

Python API for Autonomous Loops

Use gate() as a programmatic regression gate inside agent frameworks, autonomous coding loops, or CI scripts:

from evalview import gate, DiffStatus

# Full evaluation
result = gate(test_dir="tests/")
if not result.passed:
    for d in result.diffs:
        if not d.passed:
            delta = f" ({d.score_delta:+.1f})" if d.score_delta is not None else ""
            print(f"  {d.test_name}: {d.status.value}{delta}")

# Quick mode — no LLM judge, $0, sub-second
result = gate(test_dir="tests/", quick=True)

Auto-Revert on Regression

from evalview.openclaw import gate_or_revert

# In an autonomous coding loop:
make_code_change()
if not gate_or_revert("tests/", quick=True):
    # Change was automatically reverted
    try_alternative_approach()

Warning: gate_or_revert runs git checkout -- . when a regression is detected, discarding uncommitted changes. Commit or stash work-in-progress before entering the loop. You can also pass a custom revert command: gate_or_revert("tests/", revert_cmd="git stash").

MCP Integration

EvalView exposes 8 tools via MCP — works with Claude Code, Cursor, and any MCP client:

claude mcp add --transport stdio evalview -- evalview mcp serve

Tools: create_test, run_snapshot, run_check, list_tests, validate_skill, generate_skill_tests, run_skill_test, generate_visual_report

After connecting, Claude Code can proactively check for regressions after code changes:

• "Did my refactor break anything?" triggers run_check
• "Save this as the new baseline" triggers run_snapshot
• "Add a test for the weather tool" triggers create_test

CI/CD Integration

# .github/workflows/evalview.yml
name: Agent Regression Check
on: [pull_request, push]
jobs:
  check:
    runs-on: ubuntu-latest
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: pip install "evalview>=0.5,<1"
      - run: evalview check --fail-on REGRESSION

--fail-on REGRESSION gates on score drops only. For stricter gating that also catches tool sequence changes, use --fail-on REGRESSION,TOOLS_CHANGED or --strict (fails on any change).

Test Case Format

name: refund-flow
input:
  query: "I need a refund for order #4812"
expected:
  tools: ["lookup_order", "check_refund_policy", "issue_refund"]
  forbidden_tools: ["delete_order"]
  output:
    contains: ["refund", "processed"]
    not_contains: ["error"]
thresholds:
  min_score: 70

Multi-turn tests are also supported:

name: clarification-flow
turns:
  - query: "I want a refund"
    expected:
      output:
        contains: ["order number"]
  - query: "Order 4812"
    expected:
      tools: ["lookup_order", "issue_refund"]

Best Practices

• Snapshot after every intentional change. Baselines should reflect intended behavior.
• Use --preview before snapshotting. evalview snapshot --preview shows what would change without saving.
• Quick mode for tight loops. gate(quick=True) skips the LLM judge — free and fast for iterative development.
• Full evaluation for final validation. Run without quick=True before deploying to get LLM-as-judge scoring.
• Commit .evalview/golden/ to git. Baselines should be versioned. Don't commit state.json.
• Use variants for non-deterministic agents. evalview snapshot --variant v2 stores alternate valid behaviors (up to 5).
• Monitor in production. evalview monitor catches gradual drift that individual checks miss.

Installation

pip install "evalview>=0.5,<1"

Package: evalview on PyPI

Attribution and license

MIT License

Copyright (c) 2026 Affaan Mustafa

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.