Cosmos 3 Edge - NVIDIA's Open-Source 4B-Parameter World Model

What Is Cosmos 3 Edge

Cosmos 3 Edge is a 4B-parameter open-source world model released by NVIDIA, designed for real-time inference in robotics and edge AI. The model is based on the Nemotron architecture and combines an autoregressive and a diffusion dual Transformer tower, sharing a multimodal attention layer. It can run locally on devices such as Jetson Thor, RTX GPUs, and DGX, enabling 15 Hz real-time control. Developers can adapt the model to a specific robot or environment in about one day, perceiving, reasoning, and generating action policies without relying on the cloud.

Main Features of Cosmos 3 Edge

• World State Understanding: Processes visual and text tokens through the autoregressive tower, parsing objects, spatial relationships, and scene semantics in the edge environment in real time, providing a shared world representation for subsequent reasoning.
• Future State Prediction: Uses the diffusion tower to denoise and generate visual, audio, and action tokens, simulating the visual result of executing a certain action.
• Action Policy Generation: Uniformly encodes different embodiments such as robotic arms, vehicles, and cameras into compact geometric vectors of translation, rotation, and manipulation states, generating 32 continuous actions per inference and outputting control commands at 15 Hz.
• On-Device Real-Time Inference: Achieves memory-efficient, high-throughput inference on memory-constrained devices such as Jetson Thor, RTX PRO, and Jetson T2000/T3000, enabling closed-loop operation without the cloud.
• Fast Domain Adaptation: Provides complete post-training scripts and DROID dataset policy checkpoints, allowing developers to fine-tune the base model to a specific robot or environment in about one day.

Technical Principles of Cosmos 3 Edge

• Dual-Tower Shared Architecture: The model consists of an autoregressive tower and a diffusion tower. The autoregressive tower uses causal attention to process visual and text tokens for scene understanding and reasoning; the diffusion tower uses bidirectional attention to process visual, audio, and action tokens for prediction, generation, and neural simulation. The two towers each have independent LayerNorm and MLP, and share a multimodal attention layer, aligning language, video, audio, and action information into a unified representation space.
• Unified Action Representation: Maps the actions of different physical systems into compact geometric vectors, uniformly encoding translation, rotation, and manipulation states, and establishing direct connections between pixel changes and physical motion, spatial relationships, and control inputs.
• World Action Model (WAM): When used as a policy, the model receives the current observation state and simultaneously outputs the action to be executed and its expected visual result, directly linking world modeling with robot policy training. Actions can flow bidirectionally in the model: it can predict the effect of an action and infer the action from the effect.
• Edge-Optimized Inference: Based on the compact 4B-parameter design of the Nemotron architecture, combined with 4-step knowledge distillation and the vLLM optimization framework, it achieves up to 25x inference acceleration while maintaining output quality, adapting to the memory and compute constraints of edge computing platforms such as the Jetson series.

Follow WeChat and reply “Open Source” to join theAI Open Source Project Community Group

How to Use Cosmos 3 Edge

• Environment Preparation: Install the NVIDIA Container Toolkit and pull the official Docker image, or configure a Conda environment to install dependencies such as PyTorch, Diffusers, and Transformers.
• Download the Model: Download from Hugging Face nvidia/Cosmos3-Edge weights, while also obtaining post-training scripts and DROID dataset policy checkpoints.
• Post-Training Fine-Tuning: Using a small H100 cluster or DGX Station, based on the open-source Cosmos Framework, perform approximately one day of adaptation fine-tuning for a specific robot, sensor, or environment.
• Deployment Inference: Deploy the fine-tuned model to edge devices such as Jetson T2000/T3000, RTX PRO, or DGX, and achieve 15 Hz real-time action generation through vLLM or NIM microservices.

Core Advantages of Cosmos 3 Edge

• On-Device Real-Time Inference: A lightweight 4-billion-parameter design achieves 15 Hz real-time control on Jetson Thor, eliminating cloud latency.
• Fast Adaptation: Developers can fine-tune the foundation model to a specific robot or environment in about one day, greatly reducing deployment cycles.
• Unified Multimodal Architecture: The autoregressive tower and diffusion tower share attention layers, uniformly processing language, video, audio, and actions to achieve a "understand -> simulate -> act" loop.
• Open Ecosystem: Model weights, training scripts, post-training plans, and 4-step knowledge distillation checkpoints are all open source, supporting Hugging Face Diffusers and vLLM deployment.
• Benchmark Leadership: Among models of the same scale (4B), it ranks first in VANTAGE-Bench visual analysis, and robot policy learning reaches industry-leading levels.