evalview-agent-testing

历史版本与核实状态

这是2026-03-31已合并的历史指南，合并时间为21:13:33 UTC，合并提交为0f40fd030c222f1c56ae9ef05d5115ac9fd140a0。2026-10-18 UTC完整核对该提交中的SKILL.md及MIT许可，并保留署名。此处不确认当前分支仍有该指南、原目录安装命令仍可用或依赖兼容；文中的工具数、费用、性能及自动回退行为均为历史文档陈述，本轮未运行。

已合并原文

EvalView 智能体测试

为AI智能体执行自动回归测试。EvalView记录智能体行为快照，包括工具调用、参数、顺序和输出，再在每次修改后与基线比较，帮助在发布前发现变化。

适用时机

修改智能体代码、提示词或工具定义后
更新模型或更换提供方后
将智能体部署到生产环境前
为智能体项目配置CI/CD时
自主循环（OpenClaw、编码智能体）需要适应度函数时
智能体输出意外变化，需要确定变化位置时

核心流程

# 1. Set up
pip install "evalview>=0.5,<1"
evalview init              # Detect agent, create starter test suite

# 2. Baseline
evalview snapshot           # Save current behavior as golden baseline

# 3. Gate every change
evalview check              # Diff against baseline — catches regressions

# 4. Monitor in production
evalview monitor --slack-webhook https://hooks.slack.com/services/...

理解检查结果

状态	含义	操作
`PASSED`	行为与基线一致	可据检查结果决定发布
`TOOLS_CHANGED`	调用的工具发生变化	审查差异
`OUTPUT_CHANGED`	工具相同，输出变化	审查差异
`REGRESSION`	评分明显下降	发布前修复

自主循环中的Python API

在智能体框架、自主编码循环或CI脚本中，可用gate()设置程序化回归检查：

from evalview import gate, DiffStatus

# Full evaluation
result = gate(test_dir="tests/")
if not result.passed:
    for d in result.diffs:
        if not d.passed:
            delta = f" ({d.score_delta:+.1f})" if d.score_delta is not None else ""
            print(f"  {d.test_name}: {d.status.value}{delta}")

# Quick mode — no LLM judge, $0, sub-second
result = gate(test_dir="tests/", quick=True)

回归时自动撤销

from evalview.openclaw import gate_or_revert

# In an autonomous coding loop:
make_code_change()
if not gate_or_revert("tests/", quick=True):
    # Change was automatically reverted
    try_alternative_approach()

原文警告：gate_or_revert在检测到回归时运行git checkout -- .，会丢弃尚未提交的修改。原文要求进入循环前提交或暂存进行中的工作，也可指定自定义撤销命令，例如gate_or_revert("tests/", revert_cmd="git stash")。本页仅保留历史文档，未执行这些命令。

MCP集成

该历史指南称EvalView经MCP提供8个工具，可接入Claude Code、Cursor等MCP客户端：

claude mcp add --transport stdio evalview -- evalview mcp serve

工具标识：create_test、run_snapshot、run_check、list_tests、validate_skill、generate_skill_tests、run_skill_test、generate_visual_report。

连接后，指南给出的交互示例包括：

“我的重构是否破坏了功能？”触发run_check
“将当前状态保存为新基线”触发run_snapshot
“为天气工具添加测试”触发create_test

CI/CD集成

# .github/workflows/evalview.yml
name: Agent Regression Check
on: [pull_request, push]
jobs:
  check:
    runs-on: ubuntu-latest
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: pip install "evalview>=0.5,<1"
      - run: evalview check --fail-on REGRESSION

--fail-on REGRESSION只对评分下降设置闸门。若还要拦截工具顺序变化，可用--fail-on REGRESSION,TOOLS_CHANGED；--strict则对任意变化判为失败。

测试用例格式

name: refund-flow
input:
  query: "I need a refund for order #4812"
expected:
  tools: ["lookup_order", "check_refund_policy", "issue_refund"]
  forbidden_tools: ["delete_order"]
  output:
    contains: ["refund", "processed"]
    not_contains: ["error"]
thresholds:
  min_score: 70

原文也给出多轮测试示例：

name: clarification-flow
turns:
  - query: "I want a refund"
    expected:
      output:
        contains: ["order number"]
  - query: "Order 4812"
    expected:
      tools: ["lookup_order", "issue_refund"]

建议做法

**每次有意修改后保存快照。**基线应反映预期行为。
保存前先用--preview。evalview snapshot --preview展示拟变更内容，不保存。
紧密迭代时使用快速模式。gate(quick=True)跳过LLM评委；原文描述为免费、快速，当前成本与性能未实测。
**最终验证采用完整评估。**部署前不传quick=True，以获得LLM评委评分。
**把.evalview/golden/提交到Git。**基线应纳入版本管理；不要提交state.json。
**非确定性智能体可用变体。**原文称evalview snapshot --variant v2可保存有效行为变体，最多5个。
**在生产环境监测。**原文用evalview monitor发现单次检查可能漏过的渐进漂移。

安装

pip install "evalview>=0.5,<1"

软件包：PyPI上的evalview

原文元数据

name: evalview-agent-testing
description: Regression testing for AI agents using EvalView. Snapshot agent behavior, detect regressions in tool calls and output quality, and block broken agents before production.
origin: ECC
tools: Bash, Read, Write

原作者与许可

MIT License

Copyright (c) 2026 Affaan Mustafa

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.