What is Cosmos-Reason1
Cosmos-Reason1 is a series of multimodal large language models launched by NVIDIA. The models can generate responses grounded in physical reality. Cosmos-Reason1 includes two models: Cosmos-Reason1-7B and Cosmos-Reason1-56B. The models are trained in four stages: visual pretraining, general SFT, physical AI SFT, and reinforcement learning. Combining video input and text prompts, they can output responses with long reasoning chains and perform excellently on physical commonsense and embodied reasoning benchmarks, significantly outperforming other similar models. The models define an ontology for physical commonsense and embodied reasoning, construct corresponding benchmarks, and evaluate the physical AI reasoning capabilities of multimodal LLMs.
Main features of Cosmos-Reason1
- Physical commonsense understanding: Understands basic knowledge of the physical world, such as space, time, and fundamental physical laws, and judges the plausibility of events.
- Embodied reasoning: Based on physical commonsense, generates reasonable decisions and action plans for embodied agents (such as robots and autonomous vehicles).
- Long chain-of-thought: Generates detailed reasoning processes based on long chain-of-thought reasoning, improving the transparency and interpretability of decisions.
- Multimodal input processing: Supports video input, combines visual information and language instructions for reasoning, and generates natural language responses.
Technical principles of Cosmos-Reason1
- Hierarchical ontology: Defines a hierarchical ontology of physical commonsense, covering the three main categories of space, time, and fundamental physics, further subdivided into 16 subcategories.
- Two-dimensional ontology: Designs a two-dimensional ontology for embodied reasoning, covering four key reasoning capabilities of five types of embodied agents.
- Multimodal architecture: Based on a decoder-only multimodal architecture, input video is processed by a visual encoder, aligned with text token embeddings, and fed into the LLM.
- Four training stages of the model：
- Visual pretraining: Aligns the visual and text modalities.
- General supervised fine-tuning (SFT): Improves the model's performance on general vision-language tasks.
- Physical AI SFT: Uses specialized data to enhance physical commonsense and embodied reasoning capabilities.
- Physical AI reinforcement learning (RL): Further optimizes the model's reasoning capabilities based on rule-based rewards.
- Reinforcement learning: Designs a rule-based reward mechanism based on multiple-choice questions, and uses reinforcement learning to improve the model's performance on physical commonsense and embodied reasoning tasks.
Project address of Cosmos-Reason1
- Project official website：https://research.nvidia.com/labs/dir/cosmos-reason1/
- GitHub repository：https://github.com/nvidia-cosmos/cosmos-reason1
- HuggingFace model library：https://huggingface.co/collections/nvidia/cosmos-reason1
- arXiv technical paper：https://arxiv.org/pdf/2503.15558
Application scenarios of Cosmos-Reason1
- Robotic manipulation: Helps robots understand task goals, generate operation plans, and complete complex actions such as grasping and assembly.
- Autonomous driving: Processes road videos, predicts traffic dynamics, and generates safe driving decisions, such as avoidance and lane changing.
- Intelligent surveillance: Monitors abnormal behaviors in videos in real time, such as people falling or equipment failures, and issues alerts promptly.
- Virtual reality (VR)/augmented reality (AR): Based on virtual environment input, generate interactive responses to enhance user immersion.
- Education and Training: Use video explanations of physical phenomena or operational processes to assist teaching and vocational skills training.