What is DrivingDojo
DrivingDojo is a dataset jointly launched by the Institute of Automation, Chinese Academy of Sciences and the Meituan autonomous vehicle team, used for training and researching complex interactive world models for autonomous driving. The dataset contains 18,000 video clips, covering complete driving operations, multi-agent interactions, and rich open-world driving knowledge, providing a solid foundation for developing next-generation autonomous driving models. DrivingDojo defines the Action Instruction Following (AIF) benchmark, used to evaluate the ability of world models in future prediction under action control.
Main Features of DrivingDojo
- Interactive World Model Training: DrivingDojo provides a platform for training interactive world models that understand and simulate complex driving dynamics.
- Action Instruction Following (AIF): The dataset defines the AIF benchmark to evaluate the ability of world models to generate future predictions following action instructions.
- Diverse Driving Operations: Based on the DrivingDojo-Action subset, the model learns diverse driving operations such as acceleration, deceleration, emergency braking, and lane changes.
- Multi-Agent Interaction: With the DrivingDojo-Interplay subset, the model can understand and predict complex interactions between vehicles and other road users.
- Open-World Knowledge: With the DrivingDojo-Open subset, the model learns to handle rare events and long-tail scenarios in the open world.
- Video Generation and Prediction: The model generates future videos based on initial frames and action instructions, simulating and predicting driving scenarios.
Technical Principles of DrivingDojo
- Data Collection: Video data is collected using the autonomous vehicle platform of the Meituan autonomous vehicle team,.
- Data Curation: From the large amount of collected data, video clips containing rich driving operations, multi-agent interactions, and open-world knowledge are carefully selected.
- Action Encoding: Driving actions (such as acceleration, braking, steering) are encoded into a format that the model can understand, facilitating their use in video generation.
- Video Generation Model: Based on technologies such as Stable Video Diffusion, a model is developed that generates videos from initial frames and action instructions.
- Action Instruction Following (AIF): The consistency between the generated video and the input action instructions is compared to evaluate the prediction accuracy of the model.
- Evaluation Metrics: Metrics such as FID (Frechet Inception Distance) and FVD (Frechet Video Distance) are used to evaluate the visual quality of generated videos, and AIF error is used to evaluate the ability to follow action instructions.
Project Address of DrivingDojo
- Project Website：drivingdojo.github.io
- GitHub Repository：https://github.com/Robertwyq/Drivingdojo
- HuggingFace Model Library：https://huggingface.co/datasets/Yuqi1997/DrivingDojo
- arXiv Technical Paper：https://arxiv.org/pdf/2410.10738
Application Scenarios of DrivingDojo
- Autonomous Driving Algorithm Development: Researchers develop and test perception, prediction, and decision-making algorithms for autonomous vehicles.
- World Model Training: The dataset is used to train interactive world models that simulate real-world driving environments, enabling the model to predict future states and possible events.
- Multi-Agent Interaction Simulation: Simulate the interactions between vehicles and other road users (such as pedestrians, bicycles, and other vehicles) to optimize the interaction strategies of autonomous vehicles.
- Rare event and long-tail scenario handling: Train models to recognize and respond to rare events, such as animals suddenly crossing the road or obstacles on the road, improving the safety of autonomous driving systems.
- Virtual testing and validation: Test autonomous driving systems in virtual environments, reducing the cost and risk of real-vehicle testing and accelerating the development process.