What is HMA
HMA (Heterogeneous Masked Autoregression) is an open-source method developed by MIT, Meta, and the University of Illinois Urbana-Champaign for modeling the dynamics of robot action videos. HMA is based on heterogeneous pretraining, using observation and action sequences from different robot embodiments, domains, and tasks, combined with masked autoregressive techniques to generate video predictions. HMA supports both discrete and continuous variants, used respectively for fast generation and high-fidelity generation, handles the heterogeneity of action spaces, including different action frequencies, dimensions, and action spaces, and is based on a modular network architecture to achieve efficient real-time interaction. HMA has broad application prospects in robot learning, including video simulation, policy evaluation,synthetic datageneration, and use as an imitation policy, and demonstrates outstanding performance in scalability and real-time capability.
Main Features of HMA
- Video Simulation: Generates high-quality video sequences, simulates the effects of robot actions in different environments, and is used for interaction and testing in virtual environments.
- Policy Evaluation: As a high-fidelity simulator, it evaluates the performance of robot policies and predicts how policies will perform in real environments.
- Synthetic Data Generation: Generates large amounts of synthetic data, augments robot training datasets, and improves the generalization ability of policies.
- Imitation Policy: Directly serves as a policy for imitation learning, predicting robot actions under given observations.
Technical Principles of HMA
- Heterogeneous Pretraining：
- Data Sources: Pretraining uses large amounts of observation and action sequence data from different robot embodiments, tasks, and domains, covering action spaces from simple to complex.
- Handling Action Heterogeneity: By designing specific action encoders and decoders for each domain, different action spaces are mapped into a shared latent space, handling the heterogeneity of action frequencies, dimensions, and action spaces.
- Modular Architecture: The network architecture includes multiple action input modules ("stem") and action output modules ("head"), as well as a shared core spatiotemporal transformer ("trunk"), supporting efficient pretraining and flexible expansion.
- Masked Autoregression：
- Masked Objective: During training, the model randomly masks some tokens based on a masked autoencoding objective and predicts the masked parts based on the unmasked tokens, learning the joint distribution of the sequence.
- Autoregressive Generation: During inference, the model progressively unmasks and generates future video frames and action sequences. It is both efficient and able to maintain high generation quality.
- Two Variants: HMA supports a discrete variant (generating vector-quantized tokens) and a continuous variant (generating soft tokens), used respectively for fast generation and high-fidelity generation.
HMA Project Address
- Project Official Website：https://liruiw.github.io/hma/
- GitHub Repository：https://github.com/liruiw/HMA
- HuggingFace Model Library：https://huggingface.co/liruiw/hma-base-disc
- arXiv Technical Paper：https://arxiv.org/pdf/2502.04296
- Online Experience Demo：https://huggingface.co/spaces/liruiw/hma
Application Scenarios of HMA
- Real-time Video Simulation: Quickly generates videos of robot actions in different environments, used for virtual interaction testing, verifying policy effectiveness, and saving actual deployment costs.
- Policy Evaluation: As a high-fidelity simulator, evaluate robot policy performance, predict how policies will perform in real environments, and assist policy optimization.
- Synthetic data generation: Generate large amounts of synthetic data, expand the training dataset, and improve policy generalization, with particularly notable results when data is scarce.
- Imitation learning: Serve directly as an imitation policy, predict robot actions based on current observations, respond quickly to environmental changes, and improve task execution efficiency.
- Long-term planning and control: HMA supports generating long-sequence video and action predictions, helping robots perform long-term planning and model predictive control, and improving the completion rate of complex tasks.