rock-eval

Rock Eval — AI Agent Regression Evaluation

This skill uses rockcli(alias rc) and regression.py to orchestrate the complete regression evaluation workflow, covering: dispatching tasks, monitoring progress, analyzing results, diagnosing failures, and retrying.

Use cases

Run agent evaluation / benchmark / regression
View evaluation results, pass rate, reward scores
Troubleshoot or retry failed tasks
Involving rc、rockcli、regression.py, experiment ID, or results JSON file

Quick decision tree

用户想做什么？
│
├─ 跑新的回归评估 ────────────────────→ Section 1: Run
├─ 查看结果 / 生成报告 ──────────────→ Section 2: Report
├─ 任务卡在 "dispatched" ────────────→ Section 3: Sync（Monitor: Cron 每 3 分钟 sync+判可疑，可疑时调 rock-agent-debug，见 references/team-orchestration.md § Monitor）
├─ 了解任务为什么失败 ────────────────→ Section 4: Diagnose
├─ 重跑失败任务 ──────────────────────→ Section 5: Retry
├─ 深度分析失败原因 ──────────────────→ Section 6: Analyze（→ references/deep-analysis.md）
├─ 在沙箱内执行回归 / 用 zb-a 沙箱长跑 rock bench
│   ───────────────────────────────→ Section 7: Sandbox Run
├─ 手动 rc 命令 ──────────────────────→ 读 references/rockcli-cheatsheet.md
├─ Harbor Job config YAML / agent kwargs → 读 references/harbor-config-manual.md
├─ 全量回归（长跑 + 多失败分诊）
│   → TeamCreate（推荐）: references/team-orchestration-teamcreate.md
│     （TaskList 状态机 + schemas: references/schemas.json）
│   → Legacy v2（单 session）: references/team-orchestration.md（7 角色并行 pipeline）
│   → 不要混用两种模式
├─ 失败后调参 → references/team-orchestration.md § Operator（legacy）
│   或 references/team-orchestration-teamcreate.md § task-11（TeamCreate）
└─ 完整 SOP / 工作流参考 ────────────→ 读 references/sop.md

Core workflow

run  ──→  report  ──→  sync（如需）──→  diagnose  ──→  retry
 ^                                                       │
 └───────────────────────────────────────────────────────┘

The script is located in this skill directory at scripts/regression.py。Run it via an absolute path; do not copy it. Output (results/、logs/、configs/) is written to the working directory at invocation time.

1. Run — Dispatch regression tasks

python3 regression.py run \
  --bench <BENCH> \
  --dataset <DATASET> \
  --split <SPLIT> \
  --agent <AGENT> \
  --concurrency <N> \
  --window-size <N>

Required parameters

Parameter	Description
`--bench`	Bench template — run `rc agent run --help` to view currently available values
`--agent`	Agent name — run `rc agent run --help` to view currently available values
`--window-size`	Global concurrency limit(sliding window, always keeping N tasks in execution).Must be confirmed with the user before dispatch. `0` = unlimited
`--concurrency`	`--window-size` alias; when both are specified, the smaller value is used

--dataset and --split are required unless --tasks。

Resolve the task list when only a bench name is given

When the user provides only a bench name:

rc agent bench getconfig <BENCH> --raw → read the datasets fields (name, registry.split, task_names)
rc datasets <NAME> tasks --split <SPLIT> → pass to regression.py run or specify via --tasks Specify
Fallback: if the tasks subcommand returns nothing, use the --raw in the task_names

Agent / bench values

Do not hardcode — run rc agent run --help to view currently supported values. Use rc agent deps sync benchhub to refresh the bench template.

Two stable baselines:

Agent	Meaning
`oracle`	Upper-bound baseline — submit correct answers; validate the scoring pipeline
`nop`	Lower-bound baseline — do nothing; validate dispatch/image/cluster

⚠️ Before running a full regression with the real agent, ask the user whether to first use oracle/nop for smoke validation(a small number of --tasks + low concurrency). For the workflow, see references/sop.md。

Alignment baseline (score alignment scenario, optional)

After the oracle/nop smoke test and before the full run,ask the userwhether the goal is to reproduce/align to a known score:

Yes：

Obtain the reference score (provided by the user / extracted from a leaderboard/paper)
Configure cross-checks: compare model / image / split / env vars / sampling inference parameters (temperature · top_p · thinking · max_tokens · inference timeout); flag inconsistencies
Create the baseline file:baselines/<benchmark-name>-<identifier>.json(for the format, see references/data-formats.md § Align baseline file)

No: directly enter the full run.

pass-through parameters

Do not assume default values, only pass the parameters specified by the user:--image、--cluster、--model、--ee KEY=VALUE、--set path=value、--pre/--no-pre、--namespace、--cpus、--memory、--with-companion、--config、--async-mode、--user-id、--base-url、--api-key

--model Optional; if omitted, the ROCKCLI shared model is used.

Control parameters

Parameter	Description
`--resume`	Skip tasks already marked success or error
`--tasks t1,t2,...`	Run only specified tasks
`--poll-interval`	Polling interval in seconds (default 10)
`--poll-timeout`	Maximum wait time per task in seconds (default 600)

Configuration persistence

Each time run/retry automatically snapshots the configuration to configs/<experiment-id>.json。

# 保存配置模板
python3 regression.py run --bench <BENCH> --agent <AGENT> ... --save-config ./my-template.json

# 从模板运行（CLI 参数覆盖 JSON）
python3 regression.py run --from-config ./my-template.json --concurrency 8

Parameter	Description
`--save-config <path>`	Additionally save the effective configuration to `<path>`
`--from-config <path>`	Load as the base configuration; CLI parameters override the corresponding fields in the JSON

The JSON contains the complete parameter set (bench/dataset/split/agent/all pass-through parameters/tasks/concurrency/poll), excluding the experiment id (regenerated each time). Note --config(rc's JobConfig YAML) is an existing different parameter.

Output

Result JSON:results/<experiment-id>.json
Configuration snapshot:configs/<experiment-id>.json
Task log:logs/<experiment-id>/<task-id>.log

2. Report — View results

python3 regression.py report                          # 最新实验，文本格式
python3 regression.py report <EXPERIMENT_ID>          # 指定实验
python3 regression.py report --format html --open     # HTML 仪表盘
python3 regression.py report --format json            # JSON 供脚本使用

The HTML report includes: KPI cards, donut chart, reward histogram, exception grouping table, interactive task table.

3. Sync — Refresh stale status

python3 regression.py sync [EXPERIMENT_ID]
python3 regression.py sync --dry-run    # 预览，不写入
python3 regression.py sync --force      # 重新同步所有任务

After a run is interrupted, be sure to run sync before generating the report.

Monitor inspection mechanism (for long runs)

See references/team-orchestration.md § Monitor. The inspection role uses Cron every 3 minutes session-only scheduled:

sync → 2. report --format json → 3. Maintain logs/<EXP_ID>/monitor-state.json → 4. Judge "fake execution" by 4 criteria: dispatched not decreasing for ≥3 consecutive times / error accumulation / pass rate far below expectation / single task exceeding inference timeout by 1.5 times
Only when suspiciouscall rock-agent-debug(experiment_id + job_name) to confirm; if normal, do not call
After the run ends, use CronDelete to clean up the Cron

⚠️ Cron is session-only, and stops when the session is closed. For long runs across sessions, use durable: true。

4. Diagnose — Failure triage

# 概览
python3 regression.py diagnose [EXPERIMENT_ID]
python3 regression.py diagnose --status error
python3 regression.py diagnose --exception RuntimeError

# 单任务深入
python3 regression.py diagnose --task <TASK_ID>
python3 regression.py diagnose --task <TASK_ID> --remote       # + 服务端日志
python3 regression.py diagnose --task <TASK_ID> --trajectory   # + 执行轨迹
python3 regression.py diagnose --task <TASK_ID> --artifacts    # + 输出文件
python3 regression.py diagnose --task <TASK_ID> --tail 50      # 最后 50 行

5. Retry — Rerun failed tasks

python3 regression.py retry [EXPERIMENT_ID] \
  --bench <BENCH> --agent <AGENT> --concurrency <N> --window-size <N>

python3 regression.py retry --filter error ...
python3 regression.py retry --filter error --exception-type RewardFileNotFoundError ...
python3 regression.py retry --tasks t1,t2,t3 ...

	`run --resume`	`retry`
Skip	success + error	Only success
Experiment ID	Reuse original ID	New ID (with `retry_of` reference)
Filter	None	By status, exception type, manual list

retry supports the same run as --save-config/--from-config configuration persistence (JSON as base, CLI overrides).

Section 6: Analyze — In-depth failure analysis

Perform systematic trajectory analysis on completed experiments, locating the root cause of failure task by task.Complete guideRead references/deep-analysis.md。

Phase	What to do	Output
1. Experiment overview	Use `live-score.py` to obtain the full data	`overview.json`
2. Parallel in-depth analysis	Each batch of 5-8 sub-agents analyzes the trajectory of failed jobs	One per task `.md` analysis file
3. Pattern aggregation	Summary of failure category distribution + improvement suggestions	`SUMMARY.md`

# Phase 1
python3 scripts/live-score.py -e <EXP_ID> [--pre] --text
# 或
python3 scripts/fetch-overview.py <EXP_ID> [--pre] --output /tmp/bench-analysis-<EXP_ID>/overview.json

For Phase 2/3, see references/deep-analysis.md。Failure taxonomysee references/failure-taxonomy.md(8 categories + boundary judgment guide).

Section 7: Sandbox Run — execute regression inside the sandbox

Applicable to scenarios that require intranet environment access or long-term isolated execution.

Architecture: first start a "control sandbox" to install rockcli / regression.py, then distribute from within the sandbox rc agent run. The cluster where the control sandbox resides and the cluster where the agent job runs are two layers of configuration and must be confirmed separately:

Layer	Parameter location	Description
Control sandbox cluster	`rc --cluster <sandbox-cluster> sandbox start ...`	Used to install rockcli, save scripts, and run the regression control process in the background
agent job cluster	`regression.py run --cluster <job-cluster>`	Passed through to each `rc agent run`inside the sandbox, determining which ROCK cluster the evaluation job runs on

Recommended cluster combinations

Scenario	Control sandbox	agent job
zb-a regression rock bench	`zb-a`	Confirm separately according to bench/image requirements, commonly `vpc-zb-a` or a user-specified cluster
sg-a compatibility path	`sg-a`	`vpc-sg-a`

⚠️ When the user explicitly says "use the zb-a sandbox", the control sandbox must be zb-a, do not write it as vpc-zb-a。 --cluster is a rc global option; when starting the control sandbox it must be placed after rc and beforesandbox :rc --cluster zb-a sandbox start .... The agent job's --cluster is set in regression.py run / configuration JSON, and cannot be replaced by the control sandbox cluster.

Confirm with the user before execution: employee ID (user_id）、ROCK_TOKEN、ANTHROPIC_API_KEY, other env vars, rockcli version (stable/beta).

Step 1 — Start the sandbox

# zb-a 控制沙箱；auto-clear 5 天适合长跑回归，按用户要求调整
rc --cluster zb-a sandbox start \
  --auto-clear 432000 \
  --wait-for-alive \
  --memory 64g \
  --cpus 16

sandbox start only supports:--image、--memory、--cpus、--timeout、--auto-clear、--wait-for-alive. The output includes SANDBOX_ID(e.g. sb-xxxxxxxx）。

Step 2 — Install rockcli

rc sandbox <SANDBOX_ID> exec 'bash -c "$(curl -fsSL http://xrl.alibaba-inc.com/install.sh)"'
# beta 版
rc sandbox <SANDBOX_ID> exec 'bash -c "$(curl -fsSL http://xrl.alibaba-inc.com/install_beta.sh)"'
rc sandbox <SANDBOX_ID> exec 'export PATH=/root/.local/bin:/root/.nvm/versions/node/v22.23.1/bin:$PATH && rc --version'

If rc / rockcli is not in PATH, prefer to set it explicitly in the startup script:

export PATH=/root/.local/bin:/root/.nvm/versions/node/v22.23.1/bin:/usr/local/bin:/usr/bin:/bin:$PATH

Step 3 — Inject credentials

⚠️ The sandbox does not support -e passing in env vars; you must write them via exec into ~/.bashrc。

rc sandbox <SANDBOX_ID> exec 'echo "export ROCK_TOKEN=<TOKEN>" >> ~/.bashrc'
rc sandbox <SANDBOX_ID> exec 'echo "export ANTHROPIC_API_KEY=<KEY>" >> ~/.bashrc'
# 验证
rc sandbox <SANDBOX_ID> exec 'source ~/.bashrc && rc agent bench list --pre'

Step 4 — Confirm the regression configuration

On the local machine, confirm the parameters according to Section 1 and generate --save-config ./my-config.json. For rock bench in particular, confirm:

bench / dataset / split / tasks
agent / model / image
agent job cluster: e.g. --cluster vpc-zb-a, or another cluster explicitly specified by the user
Resource specifications:--cpus / --memory
--pre / --no-pre
window size / concurrency

Step 5 — Create directories and upload files

rc sandbox <SANDBOX_ID> exec 'mkdir -p /workspace/scripts /workspace/results /workspace/logs /workspace/configs'
rc sandbox <SANDBOX_ID> upload --dir <local-scripts-dir> --target-path /workspace/scripts --recursive
rc sandbox <SANDBOX_ID> upload --file ./my-config.json --target-path /workspace/my-config.json

Step 6 — Start the regression in the background

⚠️ You must use setsid or nohup ... &, otherwise the process may be killed after the exec times out and disconnects. Inside a Linux sandbox, prefer setsid; local macOS usually does not have setsid, do not verify this command on the local machine.

rc sandbox <SANDBOX_ID> exec 'bash -lc "source ~/.bashrc; export PATH=/root/.local/bin:/root/.nvm/versions/node/v22.23.1/bin:/usr/local/bin:/usr/bin:/bin:\$PATH; cd /workspace; setsid python3 scripts/regression.py run --from-config /workspace/my-config.json --cluster <JOB_CLUSTER> --window-size 10 > /workspace/logs/regression.out 2>&1 < /dev/null &"'

If the sandbox does not have setsid, fall back to:

rc sandbox <SANDBOX_ID> exec 'bash -lc "source ~/.bashrc; cd /workspace; nohup python3 scripts/regression.py run --from-config /workspace/my-config.json --cluster <JOB_CLUSTER> --window-size 10 > /workspace/logs/regression.out 2>&1 < /dev/null &"'

Step 7 — Monitor progress

rc sandbox <SANDBOX_ID> exec 'tail -50 /workspace/logs/regression.out'
rc sandbox <SANDBOX_ID> log search --log-file command.log -m 30
rc sandbox <SANDBOX_ID> exec 'pgrep -af "regression.py run|rc agent run|rockcli agent run"'

Also query the platform experiment status directly:

rc agent view --pre -e <EXP_ID> -o json --limit 100

Step 8 — Retrieve results

rc sandbox <SANDBOX_ID> exec 'ls /workspace/results/'
rc sandbox <SANDBOX_ID> download --file /workspace/results/<EXP_ID>.json
rc sandbox <SANDBOX_ID> download --file /workspace/configs/<EXP_ID>.json
python3 regression.py report /path/to/<EXP_ID>.json

Step 9 — Stop the sandbox

rc sandbox <SANDBOX_ID> stop

Be sure to complete Step 8 first; after stop, files are inaccessible.

If you need to terminate an experiment that has already been distributed, first kill the background process in the control sandbox, then stop the RUNNING/PENDING sandboxes under the experiment:

rc sandbox <SANDBOX_ID> exec 'pkill -f "regression.py run|rc agent run|rockcli agent run" || true'
rc expr <EXP_ID> sandboxes stop --dry-run
rc expr <EXP_ID> sandboxes stop -y --concurrency 10 --fetch-concurrency 5

Common Pitfalls

Pitfall	Description
Writing `zb-a` as `vpc-zb-a`	When the user asks zb-a to control the sandbox,`sandbox start` use `rc --cluster zb-a ...`
Confusing the two layers of cluster	The control sandbox cluster and the agent job cluster are two different things; confirm each separately
Forgetting agent job `--cluster`	Inside the sandbox `regression.py run` still needs the user-confirmed job cluster
`--cluster` wrong position	When starting the control sandbox it is `rc --cluster zb-a sandbox ...`, and cannot be placed after the subcommand
Not adding `setsid` / `nohup &`	After exec disconnects, the process terminates
Credentials not `source ~/.bashrc`	Each exec is a new shell; you must source
rockcli is not in PATH	Explicitly export inside the sandbox `/root/.local/bin` and the nvm node bin
Not downloading results before stop	Files are inaccessible after the sandbox stops
`sandbox start` parameter passed incorrectly	`--env`/`-e` is invalid; only the 6 parameters listed above are allowed
pictor-permissions intermittent 500	Egress IP access authentication is unstable; retry is possible

Experiment ID resolution rules

Omitted — use results/ the most recently modified file in
Experiment ID — match results/{id}.json, then prefix-match results/{id}*.json
File path — use directly

Entry resources

Path	When to read
`references/sop.md`	Full SOP, typical workflows, detailed parameter reference
`references/rockcli-cheatsheet.md`	Raw `rc` commands (manual queries, dataset browsing, sandbox management)
`references/harbor-config-manual.md`	Harbor Job `config.yaml`, JobConfig fields, timeout/kwargs
`references/data-formats.md`	Result JSON structure, task fields, exception types, parsing methods
`references/team-orchestration.md`	Full regression — legacy v2 prompt-driven (single session, 7 roles)
`references/team-orchestration-teamcreate.md`	Full regression — TeamCreate mode (recommended, cross-session recovery)
`references/schemas.json`	TeamCreate structured output schemas (SmokeOutput, ConfigConfirmOutput, SmokeDecisionOutput)
`references/deep-analysis.md`	Complete guide to in-depth failure analysis (three-phase workflow); read when triggered by Section 6
`references/failure-taxonomy.md`	Definitions of the 8 failure categories, identification methods, and boundary judgment guide

Common scenarios

Run a new benchmark

Confirm: bench, dataset, split, agent, concurrency, and passthrough parameters (image/cluster/model, etc.)
Ask whether to first use oracle/nop for smoke validation(see Section 1)
regression.py run
Generate an HTML report after completion

Run was interrupted

regression.py sync → 2. regression.py report → 3. regression.py run --resume

Why the task failed

regression.py diagnose overview → 2. select a representative task from the top exception group → 3. regression.py diagnose --task <ID> --remote --trajectory → 4. suggest a fix or retry based on the root cause

Rerun only the tasks that failed in Docker

regression.py diagnose --exception RuntimeError confirm set
regression.py retry --filter error --exception-type RuntimeError ...
regression.py report view the new experiment

View available dataset/agent/bench

Read references/rockcli-cheatsheet.md, using rc datasets、rc agent bench list , etc.

Align/reproduce a published score

Confirm bench/dataset/split, obtain the reference score source
Extract per-task reward (or aggregate pass rate)
Configure cross-check: compare model / image / split / env vars / sampling inference parameters (temperature · top_p · thinking · max_tokens · inference timeout)
Create baselines/<name>.json(format see references/data-formats.md § Alignment baseline file)
Oracle/nop smoke test → regression.py run → report + alignment comparison (actual vs expected reward, config drift, top gap tasks)
Based on the conclusion decide: tune parameters and retry / accept the difference / report to the user