What Is EMMA-X
EMMA-X is a 7-billion-parameter embodied multimodal action model introduced by the Singapore University of Technology and Design, created by fine-tuning OpenVLA on grounded chain-of-thought (CoT) reasoning data. EMMA-X combines a hierarchical embodied dataset containing 3D spatial motion, 2D gripper positions, and grounded reasoning, and introduces a novel trajectory segmentation strategy that uses gripper open/close states and robot arm motion trajectories to enhance grounded task reasoning and look-ahead spatial reasoning, achieving significant performance improvements in real-world robotic tasks, especially those requiring spatial reasoning.
Main Features of EMMA-X
- Enhanced Spatial Reasoning: Based on predicting the future 2D positions and 3D motion plans of the gripper, it optimizes the robot's long-term task planning capability.
- Embodied Task Planning: The model combines visual and task reasoning to generate action strategies adapted to the environment, improving the robot's ability to execute complex tasks.
- Trajectory Segmentation: Using gripper states and robotic arm motion trajectories, it segments operation sequences into semantically similar action segments, enhancing task understanding and motion planning.
- Reduced Hallucination Issues: By combining visual images and task reasoning, it reduces errors and hallucinations in the task reasoning process.
- Hierarchical Planning Data Generation: It generates 2D gripper positions and 3D spatial motion, as well as embodied reasoning, for each operation segment, supporting the robot's decision-making process.
Technical Principles of EMMA-X
- Hierarchical Embodied Dataset: Built on the BridgeV2 dataset, it contains 60,000 robot operation trajectories, each accompanied by detailed spatial reasoning and task reasoning information.
- Look-ahead Spatial Reasoning: The model predicts the gripper's future position and motion plan, guiding the robot's immediate actions to align with long-term goals.
- Trajectory Segmentation Strategy: Using the HDBSCAN algorithm and a custom distance metric, combined with the end effector's motion trajectory and gripper state, it dynamically segments operation sequences.
- Gemini-Generated Task Reasoning: Using the Gemini model to generate subtasks and embodied reasoning for each segment, improving the accuracy of task understanding.
- EMMA-X Architecture: Based on OpenVLA adjustments, chain-of-thought training enhances spatial reasoning and scene understanding capabilities, predicting the robot's next action strategy.
EMMA-X Project Address
- GitHub Repository：https://github.com/declare-lab/Emma-X
- HuggingFace Model Library：https://huggingface.co/declare-lab/Emma-X
- arXiv Technical Paper：https://arxiv.org/pdf/2412.11974
Application Scenarios of EMMA-X
- Manufacturing Automation: Robots are used in tasks such as assembly, packaging, and quality control, improving the efficiency and flexibility of production lines.
- Logistics and Warehousing: In warehouses, it helps robots pick, transport, and sort goods, optimizing storage space and logistics processes.
- Service Industry: In catering or hotel services, robots complete complex tasks such as cooking assistance, room cleaning, and item delivery.
- Medical Assistance: In the medical field, robots perform delicate operations, such as assisting in surgery or handling samples in laboratories.
- Home Automation:Home service robots perform cleaning, item organization, and other household chores, improving the convenience of life.