PawBench - Alibaba Tongyi's General Agent Evaluation Benchmark

What is PawBench

PawBench is a general agent evaluation benchmark launched by Tongyi Lab, designed for personal assistant and Agent scenarios, incorporating both the base model and the runtime framework (Harness) into the same evaluation system. PawBench v1.0 builds an evaluation set of 150 real tasks and 4,050 test units. Through cross-evaluation of 9 models × 3 Harnesses, it can evaluate the best combination of model + Harness, helping Harness developers precisely locate issues and verify optimizations.

Main Features of PawBench

• Joint Evaluation: Incorporates both the base model and the runtime framework (Harness) into the same evaluation system to assess the joint performance of LLM × Harness.
• Real Task Set: Builds 150 real tasks and 4,050 test units, extracted and aggregated from 6 high-quality Agent evaluation sets.
• Five-Dimensional Tagging System: Each question is structurally annotated by application scenario, atomic capability, complexity (L1-L3), input modality, and runtime environment.
• Hybrid Scoring Mechanism: Combines automatic graders (rule assertions) with LLM-as-judge (semantic quality assessment), with scores ranging from 0 to 1.
• Multi-Slice Leaderboard: Supports free switching between three dimensions: Overall (150 questions), Text (124 questions), and Multimodal (26 questions).
• Full Trajectory Retention: All tasks run in Docker sandboxes, with execution trajectories, grader artifacts, and environment snapshots fully preserved.
• In-Depth Diagnostic Analysis: Supports slicing by model scale, modality, task type, skill domain, and other dimensions to pinpoint specific behavioral defects in the Harness.

Technical Principles of PawBench

• Cross-Evaluation Matrix: Constructs a three-dimensional evaluation space of 9 models × 3 Harnesses × 150 tasks, using the controlled variable method to isolate the contributions of model capability and framework capability.
• Five-Dimensional Tagging System: Structurally annotates tasks across five dimensions: application scenario, atomic capability, complexity, input modality, and runtime environment, supporting multi-dimensional slice drill-down analysis.
• Hybrid Scoring Mechanism: Combines deterministic rule assertions (file existence, field matching, exit code) with LLM-as-judge semantic assessment, balancing objectivity and flexibility.
• Docker Sandbox Isolation: Each test unit runs in an independent container, capturing execution trajectories, workspace artifacts, and environment snapshots to ensure results are reproducible and issues are traceable.
• Artifact-Level Hard Verification: Through substantive indicators such as checking whether files are written to disk, whether diffs are generated, and whether tests pass, it avoids models "faking completion".

How to Use PawBench

• Clone the Source Code: Visit the GitHub repository agentscope-ai/PawBench to clone the evaluation framework.
• Configure the Environment: Enter the API key of the model to be tested and configure the Harness runtime environment.
• Select Tasks: Choose a specific slice (Text/Multimodal) from the 150 tasks or launch the full task set.
• Run the Evaluation: Execute tasks in Docker sandboxes, and the system automatically preserves execution trajectories and artifacts.
• Get the Score: Calculate the final score using hybrid weights through the automatic grader and LLM-as-judge.
• Submit to leaderboard: Visit agentscope-ai.github.io/PawBench to submit results and view rankings.

PawBench's core advantages

• Diagnostic Harness : Through slice analysis of 4050 test units, precisely locate behavioral defects of the framework in Skill loading, path awareness, tool configuration, and other aspects.
• Real reproducibility: Runs based on a Docker sandbox, with execution traces and environment snapshots fully preserved, so failed cases can be traced back layer by layer for review.
• Zero-threshold out-of-the-box evaluation: Reproduces the default experience after a developer's first clone, without pursuing the "theoretical upper limit of having all API Keys configured".
• Continuous regression verification: After each Harness fix, slices can be rerun for scoring to verify whether the optimization truly corresponds to the problem dimension.
• Harness score gap quantification: For the first time proves that the Harness gap (up to 6.4 points) is comparable to a major model version upgrade, providing a path for smaller models to overtake。

PawBench's project address

• Project official website：https://agentscope-ai.github.io/PawBench/
• GitHub repository：https://github.com/agentscope-ai/PawBench

Comparison of PawBench with similar competitors

PawBench application scenarios

• Harness developer optimization: Use slice analysis to pinpoint specific defects in the framework regarding Skill loading, path awareness, Web tool default availability, etc.
• Model selection reference: Help users choose the best model + Harness combination based on task types such as plain text / multimodal / Skill / Web search.
• Benchmark regression testing: Re-run the benchmark after Harness iterative fixes to verify whether the optimization truly resolves the corresponding slice issues.
• Academic research: Provide a reproducible cross-evaluation benchmark for general intelligent agent research, driving the co-evolution of models and frameworks.