VitaBench - Meituan's Large Model Agent Evaluation Benchmark

What is VitaBench

VitaBench is a large model agent evaluation benchmark for complex problems released by Meituan's LongCat team. Using high-frequency life scenarios such as food delivery ordering, dining in restaurants, and travel as carriers, it builds an interactive evaluation environment containing 66 tools and designs cross-scenario comprehensive tasks, measuring agent performance from three major dimensions: deep reasoning, tool use, and user interaction. It is the first to quantitatively decompose agent tasks, build a large-scale real-environment database, introduce a real user simulator, and achieve fine-grained behavior coverage through atomic evaluation criteria (Rubric).

Main Features of VitaBench

• Building a complex task evaluation environment: Using high-frequency life scenarios such as food delivery ordering, dining in restaurants, and travel as carriers, it builds an interactive evaluation environment containing 66 tools and designs cross-scenario comprehensive tasks to simulate complex needs in real scenarios.
• Quantifying task complexity dimensions: It quantifies task complexity from three major dimensions: deep reasoning, tool use, and user interaction, measuring reasoning complexity through metrics such as observation space size, partial observability, and number of reasoning points; distinguishes tool complexity by single-scenario tasks and cross-scenario tasks; and introduces a real user simulator to measure interaction complexity.
• Achieving fine-grained evaluation: Drawing on the latest research, it decomposes task goals into a set of atomic evaluation criteria (Rubric), scans the complete dialogue trajectory through overlapping sliding windows, and judges whether a task is completed with a strict "all-or-nothing" standard, achieving more comprehensive and fine-grained behavior coverage.
• Providing open-source resources: VitaBench is fully open source, including the project homepage, paper link, code repository, and dataset, providing rich resources for researchers and developers and promoting the development and deployment of agents in real-life scenarios.

Technical Principles of VitaBench

• Multi-dimensional complexity construction: It constructs complex tasks through three major dimensions: deep reasoning, tool use, and user interaction, simulating task complexity in real-life scenarios.
• Real-environment database: It builds a large-scale real-environment database to provide agents with a partially observable environment and measure their reasoning ability in complex environments.
• User simulator: It introduces a real user simulator to simulate the behavior and preferences of different users, enabling agents to adapt to diverse user behaviors in multi-turn dialogues.
• Atomic evaluation criteria: It decomposes task goals into a set of atomic evaluation criteria (Rubric) and scans dialogue trajectories through sliding windows to achieve fine-grained evaluation of agent behavior.
• Cross-scenario task design: It designs cross-scenario comprehensive tasks to examine agents' ability to switch execution and integrate information across multiple scenarios and evaluate their comprehensive performance in real-life scenarios.

VitaBench Project Address

• Project Official Website：https://vitabench.github.io
• Github Repository：https://github.com/meituan-longcat/vitabench
• arXiv Technical Paper：https://arxiv.org/abs/2509.26490
• HuggingFace Dataset：https://huggingface.co/datasets/meituan-longcat/VitaBench

Application Scenarios of VitaBench

• Food delivery ordering: It simulates users' complex needs when ordering food delivery, such as choosing restaurants and dishes based on personal preferences, budget, and time constraints, and evaluates agents' ability to understand user needs, recommend suitable options, and complete orders in multi-turn dialogues.
• Dining in restaurants: Covering the entire process from finding restaurants and booking seats to ordering dishes and paying the bill, it examines the agent's reasoning and tool use capabilities in complex scenarios, such as recommending suitable restaurants based on user needs and handling tasks like reservations and menu inquiries.
• Travel and Transportation: Involving multiple stages such as trip planning, transportation booking, and attraction recommendations, it evaluates the agent's overall performance in cross-scenario tasks, such as integrating different tools and information resources to provide users with personalized travel plans.
• Agent Development and Evaluation: It provides researchers and developers with standardized evaluation benchmarks, helping to assess and optimize the agent's performance in complex tasks and promoting the development and application of agent technology.
• Human-Computer Interaction Research: Through real user simulators and multi-turn dialogue tasks, it studies the interaction patterns between agents and users, enhancing the agent's capabilities in natural language understanding and dialogue management.