What is EMMA
EMMA is an end-to-end autonomous driving multimodal model launched by Waymo based on the Gemini model. It can directly map raw camera sensor data to driving-specific outputs, such as planned trajectories, perceived objects, and road graph elements. EMMA represents non-sensor inputs and outputs as natural language text and uses the world knowledge of pretrained large language models to jointly process multiple driving tasks in a unified language space. EMMA has demonstrated state-of-the-art performance on nuScenes motion planning and the Waymo Open Dataset, but it has limitations, such as a limited number of image frames it can process, the lack of integration with precise 3D sensing methods, and high computational cost. The model can advance the development of autonomous driving model architectures and improve the generalization and reasoning capabilities of autonomous driving systems in complex scenarios.
EMMA's Main Features
- End-to-End Motion Planning：
- Directly generate the future trajectory of the autonomous vehicle from raw camera sensor data.
- Convert trajectories into vehicle-specific control actions, such as acceleration and steering.
- 3D Object Detection: Use cameras as the primary sensors to detect and identify surrounding objects, such as vehicles, pedestrians, and cyclists.
- Road Graph Element Recognition: Identify and construct road graphs, including key road elements such as lane lines and traffic signs.
- Scene Understanding: Understand the context of the entire scene, including temporary road blockages and other situations that affect driving.
- Multi-Task Processing: Jointly process multiple driving tasks in a unified language space and generate outputs using task-specific prompts.
- Chain-of-Thought Reasoning: Based onChain-of-thought reasoning improves the model's decision-making ability and interpretability, allowing the model to explain its reasoning when predicting future trajectories.
EMMA's Technical Principles
- Multimodal Large Language Models (MLLMs): Based on pretrained MLLMs such as Gemini, the model is trained on large-scale internet data and possesses rich "world knowledge."
- Natural Language Representation: All non-sensor inputs and outputs (such as navigation instructions, vehicle states, trajectories, and 3D positions) are represented as natural language text.
- Visual Question Answering (VQA): Reimagine driving tasks as VQA problems, leveraging Gemini's pretraining capabilities and preserving broad world knowledge.
- Autoregressive Model: Use the autoregressive Gemini model to process interleaved text and visual inputs and generate text outputs.
- End-to-End Training: Based on end-to-end training, directly generate driving actions from sensor data, eliminating the need for symbolic interfaces between modules.
EMMA's Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2410.23262
EMMA's Application Scenarios
- Urban and Suburban Driving: EMMA can handle complex urban traffic environments and suburban road conditions, providing real-time driving decisions and trajectory planning.
- Traffic Congestion and Complex Intersections: In traffic congestion or multi-intersection scenarios, EMMA can perform effective route planning and decision-making to ensure safe and efficient navigation.
- Special Weather and Lighting Conditions: EMMA can adapt to different weather and lighting conditions, such as rain, fog, or nighttime driving, while maintaining stable driving performance.
- Construction Zones and Temporary Road Closures: Based on scene understanding capabilities, EMMA can identify construction zones and temporary road closures and make corresponding driving adjustments.
- Emergency response: When encountering emergencies, such as suddenly appearing obstacles or animals, EMMA can respond quickly and take measures such as evading or slowing down.