What is AgentCLUE-ICabin
AgentCLUE-ICabin is an AI agent evaluation benchmark focused on automotive intelligent cockpit scenarios, comprehensively assessing the tool-calling capabilities of large language models in intelligent cockpits. The benchmark is built on 12 major common vehicle usage scenarios, covering diverse travel needs from daily commuting to long-distance road trips, fully aligning with the actual interaction scenarios of domestic users. The evaluation designs multi-turn interactive dialogues of 1 to 10 turns, with at least one tool called per turn, comprehensively examining the model's interaction capabilities in complex environments.
AgentCLUE-ICabin adopts an objective 0/1 evaluation mechanism, ensuring the fairness of evaluation results by comparing the consistency of called functions and the system state after execution. The toolset is divided into five major categories: travel, vehicle control, entertainment, safety, and general, covering more than 70 functions from navigation to seat adjustment. The evaluation process includes stages such as scenario collection, toolset construction, dialogue data generation, and answer verification, ensuring the scientific rigor and practicality of the evaluation.
Main Features of AgentCLUE-ICabin
-
Scenario Construction: Based on 12 major common vehicle usage scenarios, such as daily commuting, long-distance road trips, and family outings, an evaluation set is constructed, covering diverse travel situations.
-
Multi-Turn Interaction: Designs multi-turn interactive dialogues of 1 to 10 turns, with at least 1 tool called per turn, simulating the continuous dialogue needs in real cockpit usage.
-
Tool Calling: Divides intelligent cockpit tools into 5 major types: travel, vehicle control, entertainment, safety, and general, covering more than 70 functions and comprehensively covering the core functions of the intelligent cockpit.
-
Evaluation Mechanism: Adopts a 0/1 evaluation method, determining correctness by comparing the consistency of called functions and the system state after function execution, ensuring fair and objective results.
-
Data Generation: Uses large models to generate multi-turn interactive dialogue data, which is then manually verified and optimized to form accurate automotive intelligent cockpit QA pairs, providing standard samples for evaluation.
Technical Principles of AgentCLUE-ICabin
- Scenario-Driven Multi-Turn Interaction Design
-
Scenario Construction: Based on 12 major common vehicle usage scenarios (such as daily commuting, long-distance road trips, and family outings), an evaluation set close to actual usage needs is constructed. These scenarios cover the diverse needs of users in different situations.
-
Multi-Turn Interaction: Designs multi-turn interactive dialogues of 1 to 10 turns, with at least one tool called per turn. This multi-turn interaction design simulates the continuous dialogue needs of users when actually using the intelligent cockpit, examining the model's performance in complex interactions.
-
-
Tool Classification: Divides intelligent cockpit tools into 5 major types: travel, vehicle control, entertainment, safety, and general, covering more than 70 specific functions. For example:
-
Travel Service Tools: Navigation, traffic condition inquiry, gas station inquiry, etc.
-
Intelligent Vehicle Control Tools: Air conditioning control, window control, seat adjustment, etc.
-
Entertainment Service Tools: Music playback, radio listening, video watching, etc.
-
Safety Service Tools: Tire pressure monitoring, sentry mode, child lock control, etc.
-
General Tools: Seat adjustment, steering wheel adjustment, lighting adjustment, etc.
-
-
Tool Calling: The model needs to call the corresponding tools according to user instructions, and ensure the accuracy of the calls and the correctness of the execution results.
- Objective and fair evaluation mechanism
-
0/1 evaluation method: By comparing the consistency between the functions called by the model and the reference answer, as well as the changes in system state after executing the functions, a correct/incorrect judgment is made. This evaluation method is more objective and fair, avoiding the bias of subjective scoring.
-
Multi-round feedback mechanism: The model has at most 3 attempts in each round of dialogue, and the system provides error feedback based on the model's call results, allowing the model to adjust according to the feedback.
-
-
Dialogue data generation: Use large models to generate multi-round interactive dialogue data, simulating real interaction scenarios between users and the intelligent cockpit.
-
Manual verification and optimization: The generated dialogue data and answers undergo manual verification and optimization to ensure the accuracy and practicality of the data, forming precise automotive intelligent cockpit QA pairs.
-
State tracking: During multi-round interactions, the system tracks and manages changes in the cockpit state. The model needs to consider the impact of each operation on the system state to ensure the correctness of subsequent operations.
-
State comparison: During the evaluation process, the system compares the system state after the model's operation with the expected state, ensuring that the model's operation not only calls correctly but also correctly changes the system state.
Core advantages of AgentCLUE-ICabin
-
Comprehensive scenarios: Covers 12 major typical driving scenarios, such as daily commuting, long-distance self-driving, family trips with children, etc., fully aligning with the actual needs of domestic users and ensuring that the evaluation results have high practicality and reference value.
-
Interaction complexity: Designs 1 to 10 rounds of multi-round interactive dialogue, with at least one tool called in each round, simulating the continuous dialogue needs in real use, examining the model's performance in complex interactions, and enhancing the depth and breadth of the evaluation.
-
Evaluation objectivity: Adopts a 0/1 evaluation mechanism, making correct/incorrect judgments by comparing the consistency of the called functions and the system state after execution, ensuring that the evaluation results are objective and fair and avoiding interference from subjective factors.
-
Tool richness: Divides intelligent cockpit tools into five major categories: travel, vehicle control, entertainment, safety, and general, covering more than 70 specific functions, comprehensively covering the core functions of the intelligent cockpit and providing the model with rich call options.
-
Data precision: Uses large models to generate multi-round interactive dialogue data, which is then manually verified and optimized to form precise QA pairs, ensuring the high quality and accuracy of the evaluation data and providing a reliable basis for model training and evaluation.
Application scenarios of AgentCLUE-ICabin
-
Daily commuting: Helps users check traffic conditions, play music, broadcast news, etc., during their commute, improving the convenience and comfort of commuting.
-
Long-distance self-driving: Provides functions such as precise navigation, seat massage, and gas station search for long-distance travel, ensuring smooth trips and driving comfort.
-
Family trips with children: Meets the travel needs of families with children, controls child locks, provides rear-seat entertainment content, searches for child-friendly facilities along the route, etc., ensuring child safety and travel convenience.
-
In-car office: Create a mobile office space, supporting Bluetooth conference calls, voice notes, in-car WiFi and other functions to meet users' in-car office needs.
-
Daily Shopping: Serving daily shopping and strolling needs, providing functions such as mall navigation, parking lot inquiries, and trunk opening, enhancing the convenience of shopping trips.
-
School Pick-up and Drop-off: Addressing the pain points of picking up and dropping off children to and from school, such as finding temporary parking spots, presetting the in-car temperature, and precise navigation to the school, optimizing the pick-up and drop-off process.