SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsmanage-evals

manage-evals

Testing & quality

View repository

Updated 10-16 11:27

About this skill

Managing Evaluations

Overview

OpenHands evaluations produce results stored on a CDN at https://results.eval.all-hands.dev/. Each run is identified by a path: {benchmark}/{model_slug}/{github_run_id}/. This skill enables triggering evaluation runs, comparing results between runs, and posting performance reports as GitHub PR comments.

Quick Start

Trigger an Evaluation

python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
    --sdk-ref <BRANCH_OR_TAG> --benchmark swebench --eval-limit 50

Compare Runs

python .agents/skills/manage-evals/scripts/manage_evals.py compare \
    "<benchmark>/<model_slug>/<run_id>/" \
    --auto-baseline

Compare and Post to PR

python .agents/skills/manage-evals/scripts/manage_evals.py compare \
    "<benchmark>/<model_slug>/<run_id>/" \
    --auto-baseline \
    --post-comment --pr <PR_NUMBER> --repo OpenHands/software-agent-sdk

Triggering Evaluations

Using the Script

# SWE-bench (default) on a PR branch
python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
    --sdk-ref my-feature-branch --eval-limit 50

# GAIA benchmark
python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
    --sdk-ref main --benchmark gaia --eval-limit 50

# With a specific model
python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
    --sdk-ref v1.16.0 --benchmark swebench --model-ids gemini-3-flash --eval-limit 50

# Multiple benchmarks (run the command multiple times)
for bench in swebench gaia; do
    python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
        --sdk-ref main --benchmark "$bench" --eval-limit 50 --reason "Multi-benchmark eval"
done

Available Benchmarks

Benchmark Description
swebench SWE-bench (default) — software engineering tasks
swebenchpro SWE-Bench Pro — harder software engineering tasks
gaia GAIA — general AI assistant tasks
swtbench SWT-bench — software testing tasks
commit0 Commit0 — commit generation tasks
swebenchmultimodal SWE-bench Multimodal — tasks with images
terminalbench TerminalBench — terminal interaction tasks
programbench ProgramBench — program-repair tasks against gold-standard test binaries

Trigger Options

Option Default Description
--sdk-ref (required) Branch, tag, or commit SHA to evaluate
--benchmark swebench Benchmark to run
--eval-limit 50 Number of instances to evaluate
--model-ids (first in config) Comma-separated model IDs from resolve_model_config.py
--tool-preset default Tool preset: default, gemini, gpt5, planning
--agent-type default Agent type: default, acp-claude, acp-codex
--instance-ids Specific instance IDs to evaluate (overrides eval-limit)
--reason Human-readable reason (shown in notifications)
--benchmarks-branch main Branch of the benchmarks repo
--eval-branch main Branch of the evaluation repo

Via PR Labels (Alternative)

Adding a label to a PR also triggers evaluations:

  • run-eval-1 — 1 instance (quick sanity check)
  • run-eval-50 — 50 instances (standard comparison)
  • run-eval-200 — 200 instances
  • run-eval-500 — 500 instances (full benchmark)

Comparing Evaluation Runs

Step 1: Find the Current PR's Eval Run

Eval runs are triggered by adding labels like run-eval-50 to a PR. The all-hands-bot posts a comment with results when complete.

Option A — From bot comments on the PR:

gh api repos/OpenHands/software-agent-sdk/issues/<PR_NUMBER>/comments \
    --jq '.[] | select(.user.login == "all-hands-bot") | .body' \
    | grep -o 'Evaluation:.*' | head -1

The evaluation name follows the format {github_run_id}-{model_slug_short} (e.g., 23775164157-claude-son). Extract the github_run_id from this.

Option B — From the "Evaluation Triggered" bot comment:

gh api repos/OpenHands/software-agent-sdk/issues/<PR_NUMBER>/comments \
    --jq '.[] | select(.body | test("Evaluation Triggered")) | .body'

This contains the SDK commit SHA. Cross-reference with daily metadata to find the run ID.

Option C — From daily metadata:

curl -s "https://results.eval.all-hands.dev/metadata/$(date -u +%Y-%m-%d).txt"

Each line is a run path. Match by benchmark and model to find the run.

Step 2: Identify the Run Path Components

A run path has three components:

  • benchmark: swebench, swebenchpro, gaia, swtbench, commit0, swebenchmultimodal, terminalbench, programbench
  • model_slug: Derived from model name with /:@. replaced by - (e.g., litellm_proxy-claude-sonnet-4-5-20250929)
  • run_id: The GitHub Actions workflow run ID from the OpenHands/evaluation repo

Step 3: Verify Results Exist

curl -sI "https://results.eval.all-hands.dev/<benchmark>/<model_slug>/<run_id>/output.report.json" | head -1

A 200 status confirms the run completed and results are available.

Step 4: Find a Baseline for Comparison

Automatic: The comparison script's --auto-baseline flag scans metadata files backward up to 14 days to find the most recent completed run with the same benchmark and model.

Manual: Inspect metadata files or other PR bot comments to identify a specific run:

# Check today's runs
curl -s "https://results.eval.all-hands.dev/metadata/$(date -u +%Y-%m-%d).txt" | grep "swebench/litellm_proxy-claude"

# Check yesterday's runs
curl -s "https://results.eval.all-hands.dev/metadata/$(date -u -d yesterday +%Y-%m-%d).txt" | grep "swebench/litellm_proxy-claude"

Step 5: Run the Comparison

python .agents/skills/manage-evals/scripts/manage_evals.py compare \
    "swebench/litellm_proxy-claude-sonnet-4-5-20250929/23775164157/" \
    --baseline "swebench/litellm_proxy-claude-sonnet-4-5-20250929/23773892085/"

Or with auto-baseline and PR comment posting:

python .agents/skills/manage-evals/scripts/manage_evals.py compare \
    "swebench/litellm_proxy-claude-sonnet-4-5-20250929/23775164157/" \
    --auto-baseline \
    --post-comment --pr 2334 --repo OpenHands/software-agent-sdk

Available Data Per Run

Each run stores files at https://results.eval.all-hands.dev/{run_path}/:

File Description
metadata/params.json Run parameters: SDK commit, PR number, model, eval_limit, triggered_by
output.report.json Aggregated results: resolved/submitted/total counts and instance IDs
cost_report.jsonl Per-instance cost data
results.tar.gz Full archive with all outputs

Dashboard

The eval monitor dashboard provides a visual view of runs:

https://openhands-eval-monitor.vercel.app/?run={benchmark}/{model_slug}/{run_id}/

Interpreting Results

  • Success rate = resolved / min(eval_limit, total_instances)
  • A 50-instance sample has natural variance of ±2-4 resolved instances between runs
  • Focus on instance-level changes (gained/lost) to understand regressions vs. noise
  • If the same set of instances is resolved, the difference is likely noise

Additional Resources

Reference Files

  • references/eval-infrastructure.md — Detailed documentation on the evaluation infrastructure, GCS paths, metadata format, and workflow triggers

Scripts

  • scripts/manage_evals.py — Standalone comparison script with auto-baseline detection and GitHub comment posting