ClawBench

ClawBench is an open-source benchmark designed to evaluate the ability of AI browser agents to complete everyday online tasks on real websites. These tasks include booking travel, ordering food, applying for jobs, writing reviews, and managing projects. The project is built by NAIL Group, and its sister project is HarnessBench.

Purpose

ClawBench is used to measure the end-to-end task success rate of AI browser agents on real websites. Through a five-layer recording pipeline and an agent evaluator, it compares each run with a human reference. In the historical V1 paper evaluation, the strongest evaluated agent completed about one third of the tasks.

Key Capabilities

• Provides two corpora, V1 and V2: V1 contains 152 everyday tasks and 143 live websites; V2 contains 129 tasks and 63 live websites. In total, 281 tasks, covering 163 live websites and 15 life categories.
• Five-layer recording pipeline: records the task execution process.
• Agent evaluator: compares each run with a human reference.
• Live leaderboard: displays scores by corpus, test harness, and scoring criteria.
• Task browser: supports searching, filtering, and browsing the 281 tasks by category.
• Provides a Hugging Face dataset: includes task definitions and complete execution traces for V1 and V2.
• Supports multiple browser runtimes and frameworks, including remote browser runtimes such as Kernel and Browserbase.
• Provides the RewardHarness self-evolving agent reward framework.

Usage

Quick Start

Install via pip:pip install clawbench-eval. You can also use pipx install clawbench-eval or python -m pip install clawbench-eval. After installation, the commands include:clawbench、clawbench-run、clawbench-batch、clawbench-rescore、clawbench-analyze、clawbench-reproduce and clawbench-harbor-adapt.

Clone the Repository to Run

If you need to modify drivers, bundle test cases, or build containers, you can clone the repository and use the root uv package entry point.

Prerequisites: Python 3.11+, uv, and a container engine (Docker or Podman). ClawBench automatically detects the installed engine; you can force the specification via export CONTAINER_ENGINE=docker or export CONTAINER_ENGINE=podman .

Using Podman on macOS requires a virtual machine: after installation, you need to run podman machine init && podman machine start.

Using Podman on Windows also requires running podman machine init && podman machine startfirst.

Configure the Model

On first startup, a local template will be created under models/ . Scoring requires a judge model, and you need to add an API key for deepseek-v4-pro .

Run Tasks

The recommended approach is to use the interactive TUI:clawbench. For pipelines, CI, or non-TTY environments, use clawbench-run or clawbench-batch.

Run a single task with a single model:clawbench-run. After the container starts, it prints a noVNC URL (such as http://localhost:6080/vnc.html), which you can open to watch the agent operate live. Results are saved in ./test-output/<model>/<harness>-<case>-<model>-<timestamp>/.

Run the entire corpus:clawbench-batch --cases-suite v2. You can add --max-concurrent N to run tasks in parallel, and--harness <name> to select the agent. Each task is scored by the deepseek-v4-pro judge, and scoring can be skipped via --no-judge .

Choose a Test Harness

The test harness is the agent scaffold that drives the browser, and the model is a separate axis. The default is openclaw. Select it via --harness <name> . For the full registry, see src/clawbench/runtime/harnesses/harnesses.yaml.

Other browser runtimes and frameworks

Supports remote browser runtimes such as Kernel and Browserbase.

Use cases

• Evaluate the everyday task completion ability of AI browser agents on real websites.
• Study agent performance on long-horizon tasks.
• Reproduce leaderboard results and validate the evaluation pipeline.
• Use the provided trajectory dataset for research and development.

Limitations

• The paper reports 153 tasks for V1 and 130 tasks for V2; two ASPCA tasks were removed after release, so the actual released corpora are 152 and 129.
• The trajectory dataset is large; it is recommended to use hf download --include "<pattern>" to pull a single model or a single task.
• Using Podman on macOS requires a virtual machine; only brew install podman is not enough.
• When using classic sudo-docker, files extracted from the container are owned by root on the host; the ClawBench driver automatically changes the ownership of test-output/ back to the user after each run, but if other container tools are running at the same time, it is recommended to use rootless Podman or rootless Docker.
• Scoring requires a judge model API key.
• The first run builds the container image (Chromium + ffmpeg + noVNC + the dependencies of the selected agent framework); subsequent runs reuse cached layers.

Other information

• Live leaderboard: claw-bench.com/leaderboard (V2 by default, two-stage scoring—interception + LLM judge). The full scoring formula is in eval/scoring.md.
• Paper: arXiv:2604.08523.
• Dataset:hf download NAIL-Group/ClawBench.
• How to contribute: add tasks or models, JSON specification + scoring criteria.
• The project has received over 1,000 GitHub stars.
• The paper has been accepted to EMNLP 2026 Findings and COLM 2026 WAB.
• Research work from multiple institutions is built on ClawBench, including Google DeepMind, Stanford, UC Berkeley, Google, Microsoft Research, Harvard, ETH Zürich, Oxford, Northwestern University, ByteDance Seed, and the Hong Kong University of Science and Technology, among others.