E-Bench - An agent evaluation benchmark launched by Tencent Hunyuan and others

E-BenchWhat it is

E-Bench is a multi-step tool-use evaluation benchmark launched by the Tencent Hunyuan team in collaboration with Tsinghua University AIR and Southeast University. Based on prototypes of three real products—Honor of Kings, QQ Music, and Tencent Meeting—it constructs a fully synthetic virtual environment, including 323 state-change tasks, 41 database tables, and over 76,000 data records. Through a dual-asymmetry design of information gaps and tool gaps, it forces agents to proactively collect hidden information and orchestrate multi-step tool calls, ultimately scoring with deterministic database state diffs, revealing the capability boundaries of current AI agents in real product scenarios.

E-BenchMain features

• Fully synthetic product environment construction: Based on graph-guided database population technology, it generates virtual worlds for the three major products with no isolated records and cross-table consistency.
• Dual-gap task generation: The task-generating model has full database access, while the model under test can only interact through restricted business tools, creating a dual asymmetry in information and tools.
• Deterministic state evaluation: Uses the state diff before and after database operations as the sole criterion, counting a pass only for exact matches, eliminating partial credit and judge variance.
• Joint cost-performance analysis: Simultaneously records the API cost per task, plots the Pareto frontier of accuracy versus cost, and evaluates model cost-effectiveness.
• Code execution extension: The E-Bench-Code version opens exec_code tools, which can strip away the two types of difficulty: orchestration mechanisms and information acquisition.

E-BenchTechnical principles

• Graph-guided database population: Starting from a relational schema, it builds a table-level dependency graph, populates root tables and downstream tables in topological order, uses a strong LLM as a constrained synthesizer, writes data only through insertion tools, requires downstream tables to query candidate entities from the current database, and finally uses deterministic scripts to verify and fix semantic errors such as temporal order and aggregate counts, ensuring the environment is complete and self-consistent.
• Generator-solver asymmetry: The task-generating Agent possesses query_sql and exec_code privileges, can view the complete database, and can perform complex computations; the Agent under test can only call business-level tools, forming an information gap (global state hidden) and a tool gap (internal computation tools removed), forcing the model to proactively explore the environment through multi-step parallel tool calls.
• Intent rewriting and rule replacement: Replaces specific values derived from the database with the rules that produce them, for example, "add all delisted songs from favorites" rather than directly giving song IDs, ensuring tasks cannot be completed independently without interacting with the environment.
• Cross-validation task generation: Each task is generated through a productization loop of viewing data -> determining the target -> modifying state -> summarizing intent, cross-validated by three different strong Agent models, and adopted only if at least two reproduce it consistently, ensuring task solvability and annotation accuracy.

How to use E-Bench

• Environment preparation: Obtain the E-Bench fully synthetic virtual environment from the paper's accompanying resources, and load the database templates and 323 evaluation tasks covering the three major product domains of Honor of Kings, QQ Music, and Tencent Meeting.
• Version selection: Choose the basic version E-Bench or E-Bench-Code according to the research goal.
• Model integration: Connect the LLM Agent under test to the evaluation framework, configure it so that it can only interact with the environment indirectly through restricted domain tools (such as searching songs, creating meetings, adding friends, etc.), and prohibit direct queries to the underlying database.
• Task Configuration: Set the number of independent runs per task and the starting condition of a brand-new database copy, and enable API call cost tracking to ensure results are reproducible.
• Run Evaluation: Start automated evaluation. After the Agent receives natural language instructions, it interacts with the virtual environment through multi-step tool calls and modifies the state, and the system automatically records the complete operation trajectory and Token consumption.
• Result Analysis: After the evaluation ends, the system outputs Pass@1 and Pass^3 accuracy based on the deterministic database state diff, and generates a joint cost-performance analysis report to help evaluate the model's cost-effectiveness.

Core Advantages of E-Bench

• Realistic Scale Provides Sufficient Distraction: Each database is filled to 76,000+ rows and 600,000+ data units, and the target entity is surrounded by a large number of similar records, so the model cannot get it right by relying on a sparse environment.
• Decoupled and Reusable Environment Tasks: One self-consistent product environment can support the generation of about a hundred tasks, with extremely low marginal cost; the environment layer is controllable and the task layer is extensible.
• Stable and Reproducible Evaluation: The fully synthetic environment is unaffected by the evolution of online services, and deterministic diff scoring eliminates the subjective bias of semantic judges.
• Clear Difficulty Stratification: The basic version tests multi-step orchestration and information gathering, while the Code version additionally tests code offloading capability for computation-intensive tasks.

E-Bench Project Address

• arXiv Technical Paper：https://arxiv.org/pdf/2607.23722

Comparison of E-Bench with Similar Competing Products

Application Scenarios of E-Bench

• AI Agent Capability Evaluation: Provides standardized, quantifiable evaluation of multi-step tool-using capability for LLM Agents, covering core dimensions such as information retrieval, state changes, and cross-entity orchestration.
• Model selection and productization decisions: Through a cost-performance Pareto chart, help enterprises make trade-offs between accuracy and API overhead, and identify cost-effective models.
• Agent training data synthesis: As a controllable and scalable source of training data, it is used to continuously improve the Agent's capabilities in multi-step tool calling and reliability.
• Code-enhanced Agent research: E-Bench-Code supports research on the benefit boundaries of code execution for different types of tasks (compute-intensive vs. reasoning and decision-making).
• Cross-domain benchmark expansion: The methodology can be transferred to more product backends and CLI scenarios, promoting the establishment of general agent evaluation standards.