ClawBench

Purpose

ClawBench is an open-source benchmark for evaluating whether AI browser agents can complete the online tasks people do every day, including booking travel, ordering food, applying for jobs, writing reviews, managing projects, managing email, and more. It runs against real websites, with the goal of providing reproducible measurement of an agent's end-to-end task success capability.

Main capabilities

• Task corpus: V1 is located in test-cases/v1/ and contains 152 everyday tasks across 143 online websites; V2 is located in test-cases/v2/ and contains 129 tasks across 63 online websites. In total, 281 tasks, 163 online websites, and 15 life categories. The paper reports 153 for V1 and 130 for V2; two ASPCA tasks were removed after publication, so the actually released corpora are 152 and 129.
• Evaluation mechanism: each run is recorded through a five-layer recording pipeline, and an agentic evaluator compares the run results with human reference trajectories to measure end-to-end task success rate.
• Scoring method: the leaderboard uses two-stage scoring. The first stage, Intercepted (deterministic interception), checks whether the final HTTP request matches the task's URL/method; the second stage, Reward, is judged by deepseek/deepseek-v4-pro and is divided into a lenient rubric ("match unless contradicted") and a strict rubric ("mismatch if ambiguous"). Ranking is first by Intercepted, then by Reward as a tiebreaker. The V1 Pass Rate comes from the original paper rubric, using a Claude Code agentic evaluation subagent under eval/agentic_eval.md to compare each run with human reference trajectories.
• Leaderboard: provides a live leaderboard showing scores by corpus, execution framework, and scoring standard. The V1 two-stage Reward will appear on the leaderboard after the V1 trajectory package is re-judged.
• Datasets: three datasets are released on Hugging Face, containing the task definitions and complete execution trajectories for V1 and V2, all downloadable with a single command. The trajectory datasets are large, and hf download --include "<pattern>" can be used to pull a single model or a single task.
• Execution framework (harness): the execution framework is the agent scaffolding that drives the browser, while the model is another dimension. The default framework is openclaw, selectable in clawbench-run or clawbench-batch via --harness <name>. The full registry is in src/clawbench/runtime/harnesses/harnesses.yaml. Harnesses such as WebBrain are already supported, along with Kernel and Browserbase as remote browser runtimes.
• Other capabilities: support for Gemini-as-judge, a random-click baseline harness, EdgeBench/SForge adapters, remote browser CDP support; released the RewardHarness self-evolving agent reward framework (reaching 47.4% on EditReward-Bench using only 100 preference demonstrations, without training a reward model).

Usage

Installation: a PyPI package clawbench-eval is provided. You can use pip install clawbench-eval, or use pipx install clawbench-eval and python -m pip install clawbench-eval. The commands after installation include clawbench, clawbench-run, clawbench-batch, clawbench-rescore, clawbench-analyze, clawbench-reproduce, and clawbench-harbor-adapt.

Quick start (from a source checkout): prerequisites are Python 3.11+, uv, and a container engine Docker or Podman. ClawBench automatically detects the installed engine; you can use export CONTAINER_ENGINE=docker or export CONTAINER_ENGINE=podman to force a specific one. Podman on macOS requires a VM: brew install podman alone is not enough; after installation you must run podman machine init && podman machine start, otherwise podman info will fail with the message Cannot connect to Podman. Podman on Windows likewise requires running podman machine init && podman machine start first. Run uv run … commands in PowerShell, WSL2, or Git Bash.

Step 1: Configure models. If installed from PyPI, run clawbench in the directory where you want the results and editable configuration to be located; on first launch it creates a local template under models/. Scoring requires a judge; before running any judged batch, you need to add an API key for deepseek-v4-pro—this is the judge used by all published leaderboard rows. The PurelyMail credentials for the one-time run mailbox are provided in the committed .env; you only need to edit .env if you want to use your own PurelyMail account or enable the optional HuggingFace upload. The first run builds the container image (Chromium + ffmpeg + noVNC + the dependencies of the selected proxy harness); subsequent runs reuse the cached layers and complete within seconds.

Step 2: Run your first task. The interactive TUI is recommended, with guided model and test case selection; it requires an interactive terminal, and for piped / CI / non-TTY scenarios please use clawbench-run or clawbench-batch directly. When running a single task against a single model, the script will print the noVNC URL after the container starts (for example http://localhost:6080/vnc.html），打开可实时观看代理操作；若 If port 6080 is occupied, an alternative port will be selected automatically. Results are located in ./test-output/<model>/<harness>-<case>-<model>-<timestamp>/, containing the complete five-layer recording. When running the entire corpus, your-model is the key configured in step 1; --cases-suite v2 runs the complete V2 corpus (it can be replaced with v1-lite to run a 20-task subset). You can add --max-concurrent N to run tasks in parallel (default 2 locally, 1 when using Kernel or Browserbase), and select an agent with --harness <name>. Each task is intercepted and scored by the deepseek-v4-pro judge from step 1; pass --no-judge to skip scoring. batch-summary.json and the recording of each run are located under ./test-output/.

When analyzing completed batches, you can use clawbench-analyze to view Stage-1/Stage-2 ratios, failure classifications, per-category results, and self-reported gaps. When generating human reference runs, you can manually open the noVNC URL to complete the task yourself and then close the tab; you can also keep the session open and let an external browser agent drive it, while ClawBench records and intercepts.

Quick start: first run test-cases/v1-lite/, which is a curated 20-task subset of V1, selected by well-known websites, real-world relevance, difficulty, and category diversity, consistent with browser-use/benchmark's convention of 20 tasks per source, and can provide a credible signal at lower cost. The tier distribution is flagship 9 / core 8 / wildcard 3, covering daily life (OpenTable, DoorDash, Instacart, TaskRabbit), entertainment (Eventbrite, Goodreads, Fandango), creation (Asana, Mailchimp, Squarespace), travel (Airbnb), education (LeetCode), development technology (GitHub), academic (Overleaf), personal management (1Password), and more. All Lite tasks, regardless of the form of url_pattern, are handled by eval/agentic_eval.md. Use --cases-suite v1-lite to run, or view the linked task files in test-cases/v1-lite/.

Task example: Task 001 requires the agent to visit ubereats.com, read the fictitious user's home address from /my-info/alex_green_personal_info.json and fill it into the delivery address field, search for “Pad Thai,” select a restaurant that can deliver Pad Thai to that address, open the item details page to find the customization or special instructions field and enter “no peanuts,” add to cart and handle the login prompt (using the one-time email credentials), and finally reach checkout and click Place Order. The agent receives this instruction verbatim and has read-only access to /my-info/alex_green_personal_info.json (fictitious user name, home address, phone, date of birth) as well as the one-time email account used for the login prompt, and has 30 minutes to trigger the POST request; the container will be killed on timeout. The interceptor captures the POST request triggered by the final Place Order click.

Applicable scenarios

Suitable for evaluating the daily task completion ability of browser AI agents on real websites, and can be used for research, leaderboard comparison, experiment reproduction, judge configuration validation, and agent harness iteration. It provides Path A (rerun the agent and then score) and Path B (skip the run and re-judge published trajectories) to reproduce the leaderboard; the official statement is that for the same model under the same judge (deepseek/deepseek-v4-pro, lenient rubric), two independent runs have a reproduction error within ±2 percentage points for Intercepted and Reward on the V2 corpus.

Limitations

Tasks have a 30-minute timeout, and the container will be killed on timeout. Podman on macOS and Windows requires initializing and starting the VM first. When using classic sudo-docker, files extracted from the container are owned by root on the host; the ClawBench driver detects this after each run and automatically returns test-output/ to the user, but if other container tools are used at the same time, rootless Podman or rootless Docker can completely avoid this issue. V1 two-stage Reward numberNote: not yet appearing on the leaderboard; runs of Codex and Claude Code on V2 and the V1 OpenClaw aggregation are still in progress. The number of tasks reported in the paper differs from the released corpus due to the removal of two ASPCA tasks.