VitaBench 2.0 - Meituan LongCat's Long-term Dynamic Agent Evaluation Benchmark

What is VitaBench 2.0

VitaBench 2.0 is the first long-term dynamic user modeling agent evaluation benchmark in real-life scenarios launched by the Meituan LongCat team. It includes 56 simulated users, 819 complex tasks, over 2,000 dynamic preferences, and 66 executable tools, with an average interaction span of 1,580 days, systematically evaluating the personalization and proactivity capabilities of large models in long-term interactions.

Main Features of VitaBench 2.0

• Simulated User Trajectory Construction: Build long-term life trajectories for 56 virtual users covering domains such as food delivery, in-store dining, and travel.
• Dynamic Preference Evolution: Embed over 2,000 preferences, with each user undergoing more than 48 dynamic changes on average.
• Complex Task Evaluation: Provide 819 executable tasks spanning the user lifecycle.
• Dual Memory Mode Showdown: Uniformly evaluate Agentic Memory (proactively maintaining profiles) and RAG Memory (retrieving historical fragments).
• Proactive Task Design: Test AI's ability to proactively ask questions rather than make blind decisions when information is insufficient.

Technical Principles of VitaBench 2.0

• Three-Dimensional Deconstruction Architecture: Connect user information (profile + preferences + history + tasks) → personalized memory → agent tasks into a complete closed loop.
• Timeline Exposure: Strictly expose interaction events to the Agent in chronological order, faithfully reproducing the evolution and drift of user preferences.
• Memory Arena Mechanism: Through an extensible interface, allow two memory architectures to compete fairly under the same user scenarios, evaluating the real impact of different designs on decision-making.
• Noise Signal Separation: About 20% of interactions contain noise such as irrelevant, exploratory, and proxy interactions, testing the model's ability to extract true preferences from mixed signals.

How to Use VitaBench 2.0

• Clone the Repository: Execute git clone https://github.com/meituan-longcat/vitabench-2.0.git to obtain the evaluation framework and run scripts.
• Download Data: Pull the standardized dataset containing 56 user trajectories, 819 tasks, and 66 tools from the HuggingFace dataset.
• Install Dependencies: Enter the project directory and run the installation commands to configure the Python environment and required libraries.
• Select Mode: Choose one of the three memory settings—Full Context, Agentic Memory, or RAG Memory—as the evaluation baseline.
• Connect Model: Follow the repository documentation to connect your large model or Agent to the evaluation interface, enabling it to receive user events in chronological order and call tools.
• Start Evaluation: Run the evaluation scripts to have the Agent execute standardized tasks sequentially in scenarios such as food delivery, in-store dining, and travel.
• View Report: Analyze the generated performance report, compare metrics such as Avg@4 and Pass@4 as well as time decay curves, and identify the model's weaknesses in long-term user modeling.

Core Advantages of VitaBench 2.0

• Industry First: The first evaluation benchmark that combines agent scenarios with a rich user ecosystem and targets long-term dynamic user modeling in real life.
• Ultra-Long Time Span: The average interaction period reaches 1,580 days (maximum 2,974 days), faithfully reproducing the long-term evolution and drift of user preferences.
• High-fidelity user profiles: 56 virtual users built on real-world statistical data, covering multidimensional attributes such as gender, age, city tier, occupation, and relationship status.
• Dynamic preference evolution: Embedded with over 2,000 preferences, each user undergoes an average of more than 48 dynamic changes, simulating habit changes in real life.
• Unified memory evaluation ecosystem: Built the first long-term agent evaluation platform in real-world scenarios, enabling unified comparison of two architectures: Agentic Memory and RAG Memory.

VitaBench 2.0 project addresses

• Project website：https://vitabench2.github.io/
• GitHub repository：https://github.com/meituan-longcat/vitabench-2.0
• HuggingFace model library：https://huggingface.co/datasets/meituan-longcat/VitaBench-2.0
• arXiv technical paper：https://arxiv.org/pdf/2605.27141

Comparison of VitaBench 2.0 with similar competitors

Application scenarios of VitaBench 2.0

• AI personal assistant evaluation: Tests intelligent assistants' user understanding and personalized service capabilities during long-term companionship.
• Intelligent customer service optimization: Evaluates the accuracy of customer service systems in remembering user preferences and dynamically adapting.
• Life service platforms: Provides a realistic evaluation environment for recommendation algorithms in scenarios such as food delivery, transportation, and hotels.
• Memory module R&D: Helps researchers compare and select the performance of Agentic Memory and RAG Memory in different scenarios.
• Exploration of long-context model boundaries: Examines the capability limits of large models in ultra-long temporal and high-noise scenarios.