What is Magma
Magma is a new multimodal AI foundation model launched by Microsoft Research that provides general capabilities for multimodal AI agents. Magma can understand and execute tasks with multimodal inputs, covering digital and physical environments. Magma is pretrained on large-scale vision-language data and action data. Magma combines language intelligence, spatial intelligence, and temporal intelligence, and can complete complex tasks ranging from UI navigation to robot manipulation. In experiments, Magma demonstrated excellent performance under both zero-shot and fine-tuning settings, surpassing existing specialized models in robot manipulation and multimodal understanding tasks.
Magma's main features
- Multimodal understanding: It can process data in multiple modalities such as images, videos, and text, and understand their semantic, spatial, and temporal information. It supports tasks ranging from simple image recognition to complex video understanding.
- Action planning and execution: It decomposes complex tasks into a series of executable action sequences. It supports everything from UI navigation (such as web operations and mobile app operations) to robot manipulation in physical environments (such as grasping, placing, and moving objects).
- Environmental adaptability: It adapts to a variety of downstream tasks in a zero-shot setting, including UI navigation, robot manipulation, and multimodal understanding.
Magma's technical principles
- Pretraining architecture: It uses a convolutional network (such as ConvNeXt) as the visual encoder to process image and video data. The encoded visual information is input together with language tokens into a large language model (LLM) to generate action sequences or language descriptions.
- Set-of-Mark (SoM): It annotates actionable visual objects in images (such as buttons in a GUI or the target position of a robot arm). By predicting the positions of these markers, it helps the model understand and perform action grounding.
- Trace-of-Mark (ToM): It annotates the motion trajectories of objects in videos (such as the motion path of a robot arm). By predicting future trajectories, it helps the model understand and plan action sequences, enhancing the understanding of temporal dynamics.
- Multimodal data fusion: The pretraining data includes images, videos, robot manipulation data, and data from multimodal understanding tasks. Based on SoM and ToM techniques, these different types of data are unified into a single pretraining framework, improving the model's generality and adaptability.
- Zero-shot and fine-tuning capabilities: The pretrained model can be directly applied to unseen tasks (zero-shot), showing strong generalization ability. After fine-tuning on a small amount of data, it can further improve performance and adapt to the needs of specific tasks.
Magma's project addresses
- Project official website：https://microsoft.github.io/Magma/
- GitHub repository：https://github.com/microsoft/Magma
- arXiv technical paper：https://www.arxiv.org/pdf/2502.13130
Magma's application scenarios
- Web and mobile app operations: Automatically complete tasks such as searching, installing apps, and filling out forms.
- Robot manipulation: Control robots to complete tasks such as grasping, placing, and moving objects.
- Video understanding: Analyze video content and answer related questions.
- Intelligent assistant: As a virtual assistant, understand instructions and complete interactive tasks.
- Education and training: Assist teaching and provide operational guidance and feedback.