RoboBrain - BAAI's Open-Source Embodied Brain Model

What is RoboBrain

RoboBrain is an open-source embodied brain model launched by BAAI, driving the transition from single-machine intelligence to swarm intelligence. It consists of three modules: a base model for task planning, an A-LoRA module for affordance perception, and a T-LoRA module for trajectory prediction. RoboBrain adopts a multi-stage training strategy, possesses long historical frame memory and high-resolution image perception capabilities, and can map abstract instructions to concrete actions. It performs excellently in evaluation tasks such as task planning, affordance perception, and trajectory prediction.

RoboBrain's Main Features

• Planning Capability: Decomposes complex operation instructions into manageable subtasks. For example, decomposing "lift the teapot and pour water into the cup" into steps such as "approach the teapot and lift it," "move the teapot so the spout aligns with the cup opening," and "tilt the teapot to pour water."
• Affordance Perception: Identifies and interprets the actionable regions of interaction objects, such as the handle or spout of a teapot.
• Trajectory Prediction: Predicts the complete trajectory required to complete an operation, such as the motion trajectory from the current position to the teapot handle.

RoboBrain's Technical Principles

• Model Architecture: RoboBrain is based on the LLaVA framework and consists of the following three main modules:
  • Visual Encoder: Uses the SigLIP model to encode input images into visual features.
  • Projector: Maps visual features to the same dimension as text embeddings through a two-layer MLP.
  • Large Language Model (LLM): Adopts the Qwen2.5-7B-Instruct model for understanding and generating text instructions.
• Multi-Stage Training Strategy: RoboBrain adopts a multi-stage training strategy to improve its performance in robotic manipulation tasks:
  • General Vision Training (OneVision Training): Pre-training on large-scale general vision datasets to develop basic visual and language understanding capabilities.
  • Robotic Task Training: Fine-tuning on the ShareRobot dataset to enhance task planning, affordance perception, and trajectory prediction capabilities.
• Dataset Support: RoboBrain's training relies on the ShareRobot dataset, which is a high-quality heterogeneous dataset containing multi-dimensional annotations such as task planning, object affordance regions, and end-effector trajectories. The diversity and accuracy of the dataset are carefully designed to support the model's performance in complex tasks.
• Inference Process: In practical applications, RoboBrain first perceives visual input, decomposes the input instructions into a series of executable subtasks, and then performs affordance perception and trajectory prediction. The step-by-step processing approach enables the model to efficiently transform abstract instructions into concrete robotic operations.

RoboBrain's Project Address

• Project Official Website：https://superrobobrain.github.io/
• Github Repository：https://github.com/FlagOpen/RoboBrain
• HuggingFace Model Library：https://huggingface.co/BAAI/RoboBrain
• arXiv Technical Paper：https://arxiv.org/pdf/2502.21257

RoboBrain's Application Scenarios

• Multi-Robot Collaboration: As the core brain model of RoboOS, a cross-embodiment embodied large-small brain collaboration framework, RoboBrain can achieve efficient collaboration among multiple different types of robots.
• Complex Task Planning: RoboBrain can decompose complex operation instructions into manageable subtasks, such as "Water plants", "Put the pot in the drawer", "Cluster blocks of the same color into different corners", and other tasks, and RoboBrain can generate detailed planning steps.
• Operable Region Perception: RoboBrain can recognize and interpret the operable regions of interaction objects. For example, in the "Cluster blocks of the same color into different corners" task, RoboBrain can recognize the operable regions of blocks of different colors and plan a reasonable operation path.
• Real-time Feedback and Optimization: RoboBrain combines the device-cloud collaboration capabilities of RoboOS to receive execution feedback in real time, dynamically adjust strategies according to environmental changes, continuously optimize task planning, and improve robustness.