What is OpenEMMA
OpenEMMA is an open-source end-to-end autonomous driving multimodal model framework jointly developed by Texas A&M University, the University of Michigan, and the University of Toronto. It is based on pretrained multimodal large language models (MLLMs) to process visual data and reason about complex driving scenarios. The framework is based on a chain-of-thought reasoning process, significantly improving the model's performance in trajectory planning and perception tasks, while integrating a specially optimized YOLO model to improve the accuracy of 3D bounding box prediction. OpenEMMA provides a platform that promotes broader research and development and advances autonomous driving technology.
Main Features of OpenEMMA
- End-to-End Trajectory Planning: Learns driving actions directly from sensor inputs, enabling end-to-end optimization from perception to decision-making without symbolic interfaces.
- Multimodal Data Processing: The framework processes forward camera images and textual historical ego vehicle states as input, framing the driving task as a visual question answering (VQA) problem.
- Chain-of-Thought Reasoning: Adopts a chain-of-thought reasoning process, guiding the model to generate detailed descriptions of key objects, behavioral insights, and meta-driving decisions.
- 3D Object Detection: By integrating a fine-tuned YOLO model, OpenEMMA can accurately detect objects on 3D roads, improving the accuracy of object detection.
- Human-Readable Output: Based on the pre-stored world knowledge of MLLMs, OpenEMMA can produce interpretable, human-readable output for perception tasks such as scene understanding.
Technical Principles of OpenEMMA
- Pretrained MLLMs: Based on pretrained MLLMs, it processes complex visual data and reasons about driving scenarios.
- Chain-of-Thought Reasoning Process: Based on chain-of-thought reasoning, the model can generate speed vectors and curvature vectors, which are used to calculate the vehicle's future trajectory.
- Speed and Curvature Vectors: Given speed and curvature vectors, the model first integrates the heading angle at each time step, then calculates the x and y components of the speed, and finally uses the integrated speed components to calculate the final trajectory.
- Object Detection Enhancement: To overcome the limitations of MLLMs in spatial reasoning, a YOLO model specially optimized for 3D bounding box prediction is integrated.
- End-to-End Planning and Reasoning: OpenEMMA follows an instruction-based approach, prompting MLLMs to produce human-interpretable knowledge, decomposing the trajectory generation task into human-interpretable components that reflect the driving process.
Project Address of OpenEMMA
- GitHub Repository：https://github.com/taco-group/OpenEMMA
- arXiv Technical Paper：https://arxiv.org/pdf/2412.15208
Application Scenarios of OpenEMMA
- Urban Road Driving: In complex urban traffic environments, it handles changing traffic signals, pedestrians, bicycles, and other vehicles, providing real-time decision-making and trajectory planning for autonomous vehicles.
- Highway Driving: On highways, it handles vehicles traveling at high speeds and provides decision support for operations such as lane keeping, overtaking, and obstacle avoidance.
- Parking and Low-Speed Driving: In parking lots or low-speed environments, it helps autonomous vehicles perform precise parking operations, avoid obstacles, and navigate in tight spaces.
- Night Driving: OpenEMMA can work under low-light conditions, providing decision support for night driving, including object detection and trajectory planning.
- Complex weather conditions: Under complex weather conditions such as rain and fog, assist autonomous vehicles in maintaining safe driving and reduce the impact of weather on driving.