What is AVD2
AVD2 (Accident Video Diffusion for Accident Video Description) is an innovative framework launched by Tsinghua University in collaboration with the Hong Kong University of Science and Technology, Jilin University, Nanjing University of Science and Technology, Beijing Institute of Technology, Fudan University, and other institutions, used for autonomous driving accident video understanding. Based on generating accident videos aligned with detailed natural language descriptions and reasoning, it significantly improves the ability to understand complex accident scenarios. AVD2 combines video generation and an accident analysis system, and can generate high-quality video content containing accident descriptions, cause analysis, and preventive measures. Based on AVD2, researchers created the EMM-AU (Enhanced Multi-Modal Accident Video Understanding) dataset, providing strong data support for accident analysis and prevention. Experimental results show that AVD2 performs excellently in both automatic evaluation metrics and human evaluation, setting a new benchmark for the safety and reliability of autonomous driving.
Main functions of AVD2
- Accident video generation: Based on advanced video generation technology, generate high-quality accident videos aligned with accident descriptions, cause analysis, and preventive measures.
- Accident cause analysis: Provide detailed causes of accidents to help understand the complexity of accidents.
- Preventive measure recommendations: Based on accident analysis, propose effective preventive measures to reduce the occurrence of similar accidents.
- Dataset enhancement: Based on generating new accident videos, expand and enrich accident video datasets (such as EMM-AU), providing stronger data support for autonomous driving safety research.
- Video understanding and reasoning: Combining natural language processing and computer vision technologies, generate descriptions and reasoning related to accident videos, improving the interpretability of accident scenarios.
Technical principles of AVD2
- Video generation technology: Use advanced text-to-video generation models such as Open-Sora 1.2, fine-tune pretrained models, and generate high-quality videos aligned with accident descriptions. Combine super-resolution technology (such as Real-ESRGAN) to improve video quality and ensure that the generated videos have high clarity and detail.
- Video understanding and description generation: Based on the ADAPT (Action-aware Driving Caption Transformer) framework, combined with the Swin Transformer and BERT architectures, achieve deep fusion of visual features and text features of accident videos. Self-critical sequence training (SCST) optimizes the description generation process and, based on a reinforcement learning mechanism, makes the generated descriptions better meet the quality standards of human evaluation.
- Accident analysis and reasoning: Based on natural language processing technology, convert the visual content of accident videos into detailed natural language descriptions, including accident causes and preventive measures. Combine accident videos and text descriptions to generate reasoning results aligned with accident scenarios, helping autonomous driving systems understand and respond to complex accident scenarios.
- Dataset enhancement and evaluation: Use generated accident videos to expand the dataset, providing richer training data for autonomous driving accident analysis. Based on automated evaluation metrics (such as BLEU, METEOR, CIDEr) and human evaluation, verify the quality of generated videos and descriptions.
AVD2 project address
- Project official website：https://an-answer-tree.github.io/
- GitHub repository：https://github.com/An-Answer-tree/AVD2
- arXiv technical paper：https://arxiv.org/pdf/2502.14801
Application scenarios of AVD2
- Autonomous driving R&D engineers: Used for developing and optimizing autonomous driving systems, analyzing accident scenarios, and improving algorithms and models.
- Traffic management departments: Help formulate traffic rules and safety policies, optimize road design, and prevent accidents.
- Automakers: Applied in the design and testing of vehicle safety systems to enhance vehicle safety performance.
- Researchers and scholars: Used in research in the fields of autonomous driving and traffic safety to explore new technologies and methods.
- Autonomous driving testers: Test the accident-handling capabilities of autonomous driving systems and verify the reliability and safety of the systems.