Matrix-Game 3.0 - Kunlun Wanwei's Real-Time Interactive World Model

What is Matrix-Game 3.0

Matrix-Game 3.0 is a real-time interactive world model launched by Kunlun Wanwei (Skywork AI), supporting real-time generation at 720p@40FPS. The model uses an error buffer mechanism for self-correction and combines camera-aware memory retrieval to ensure long-range consistency, stably generating interactive videos lasting several minutes. Matrix-Game 3.0 is trained on Unreal Engine synthetic data, AAA game data, and real videos. The 5B distilled version supports 3-step fast sampling, while the 28B MoE version further improves quality. As the foundation of the "Cat Forest Academy 2.0" platform, it aims to build an interactive AAA-level AI game world.

Main Features of Matrix-Game 3.0

• Real-Time Interactive Generation: Supports keyboard/mouse action input, generating interactive video at 720p resolution in real time at 40FPS.
• Long-Range Memory Retention: Through camera-aware memory retrieval, maintains spatiotemporal consistency of scenes and objects in minute-level long videos.
• Self-Correction Capability: The model uses an error buffer mechanism to automatically correct prediction deviations, reducing visual drift and accumulated errors.
• Multimodal Control: Supports joint conditional control of text prompts, action inputs, and camera poses.
• Efficient Inference Deployment: The model supports the 5B distilled model requiring only 3 sampling steps to run in real time, while the 28B MoE version provides higher generation quality.

Technical Principles of Matrix-Game 3.0

• Data Engine: Integrates Unreal Engine synthetic data, large-scale AAA game automated collection data, and real-world video enhancement to build an industrial-grade infinite data pipeline, producing Video-Pose-Action-Prompt quadruple training data.
• Error-Aware Foundation Model: Uses a bidirectional Diffusion Transformer to jointly model historical frames, noisy current frames, and action conditions. It records prediction residuals through an error collection mechanism and injects them into training, enabling the model to learn self-correction to bridge the gap between training and inference.
• Memory-Enhanced Generation: Introduces camera-aware memory frame retrieval on top of the foundation model, selecting relevant historical content based on camera pose and field-of-view overlap, and achieving long-range spatiotemporal consistency through a joint self-attention mechanism and Plücker geometric encoding.
• Few-Step Distillation Aligned with Training and Inference: Based on a multi-stage autoregressive distillation strategy using Distribution Matching Distillation, the student model performs multi-stage rollouts simulating actual inference to ensure consistency between training and inference distributions. Combined with INT8 quantization and VAE decoder distillation, it achieves real-time inference.

Key Information and Usage Requirements of Matrix-Game 3.0

• Release Date: March 27, 2026 (2026 Zhongguancun Forum)
• Development Team: Kunlun Wanwei / Skywork AI
• Model Positioning: Real-time streaming interactive world model
• Core Capabilities: 720p@40FPS real-time generation, minute-level long-range memory consistency
• Model Versions: Offers 5B base version / 5B distilled version / 28B MoE version
• Official Resources: GitHub / HuggingFace / Demo Site
• Hardware Requirements
  • GPU: NVIDIA A/H series (supports single-card or multi-card inference)
  • System：Linux
  • Memory：64GB RAM

Core Advantages of Matrix-Game 3.0

• Industrial-Grade Real-Time Performance: The 5B distilled model can achieve real-time generation at 720p@40FPS, breaking through the speed bottleneck of interactive video generation and reaching an industrial standard that can be actually deployed.
• Long-range consistency breakthrough: It pioneers an error buffer and camera-aware memory retrieval mechanism, solving the drift problem in long video generation with diffusion models and supporting stable minute-level output.
• Training-inference alignment: The multi-segment autoregressive distillation strategy ensures consistency between training and inference distributions. 3-step sampling can achieve the quality of a 50-step base model, improving efficiency by 16 times.
• Leading data engine: It integrates Unreal Engine synthesis, AAA game capture, and real video enhancement to build an unlimited high-quality data pipeline.

How to use Matrix-Game 3.0

• Environment preparation: Configure a Python 3.12 conda environment on a Linux system and install the FlashAttention dependency library.
• Repository installation: Clone the GitHub repository locally and run pip to install all dependencies in requirements.txt.
• Model download: Use the huggingface-cli command to download the pretrained model weight files from HuggingFace.
• Input preparation: Prepare a starting image and a text prompt describing the scene content as generation conditions.
• Basic inference: Run the torchrun command to start generation, setting 704×1280 resolution, INT8 quantization, and 3-step sampling to achieve real-time generation.
• Parameter adjustment: Choose the base model (50-step high quality) or the distilled model (3-step fast) according to your needs, and enable interactive mode to customize action input.

Project address of Matrix-Game 3.0

• Project official website：https://matrix-game-v3.github.io/
• GitHub repository：https://github.com/SkyworkAI/Matrix-Game/tree/main/Matrix-Game-3
• HuggingFace model library：https://huggingface.co/Skywork/Matrix-Game-3.0
• Technical paper：https://github.com/SkyworkAI/Matrix-Game/blob/main/Matrix-Game-3/assets/pdf/report.pdf

Analysis of similar competitors to Matrix-Game 3.0

Application scenarios of Matrix-Game 3.0

• AI game development: As the foundation of the "Cat Forest Academy 2.0" platform, it builds an interactive AAA-level AI game world, enabling real-time dynamic scene generation and player action response.
• Virtual world construction: The model can create minute-level coherent open-world environments, support free exploration from first-/third-person perspectives, and is suitable for the metaverse and virtual simulation.
• Real-time interactive entertainment: It supports real-time keyboard-and-mouse control, generates smooth interactive video content, and is applied to AI-driven interactive storytelling and immersive experiences.
• Game prototyping: It rapidly generates diverse game scenes and dynamic effects, accelerating proof of concept and visual previews in the early stages of game development.