What is Matrix-Game
Matrix-Game is Kunlun Wanwei's open-source industry's first 10B+ spatial intelligence large model, and it isMatrix-Zeroan interactive video generation large model within a world model. The model is based on a two-stage training strategy, generating coherent and controllable interactive videos according to user input. It possesses advantages such as fine-grained user interaction control, high-fidelity visual and physical consistency, and multi-scenario generalization capabilities. It can be used in fields such as virtual game world construction, film and television, and metaverse content production, setting a new benchmark for building a universal virtual world foundation.
Main Features of Matrix-Game
- Controllable Video Generation: Users can freely explore, manipulate, and even create richly detailed virtual worlds with reasonable physical rules based on simple keyboard commands, mouse movements, and other operations.
- Multi-Scenario Generalization: Possesses generalization capability across various Minecraft game scenarios (such as forests, beaches, deserts, glaciers, etc.), with the potential to generalize to non-Minecraft game environments.
- Autoregressive Long Video Generation: Supports autoregressive long video generation, achieving smooth transitions between actions and viewpoints, ensuring temporal consistency and environmental adaptability.
- Systematic Evaluation: Proposes a unified GameWorld Score standard, comprehensively quantifying model performance from four dimensions: video visual quality, temporal quality, action controllability, and physical rule understanding.
Technical Principles of Matrix-Game
- Two-Stage Training Strategy: Uses large-scale unlabeled Minecraft game video data to pretrain the model to learn basic environmental features and dynamic patterns. Uses Minecraft and Unreal controllable video data with keyboard and mouse control signals for fine-grained controllable training, enabling the model to generate corresponding interactive videos based on user input.
- Image-to-World Modeling: A single reference image serves as the starting point for generating interactive videos, without relying on language prompts, modeling spatial geometry, object motion, and their physical interactions based on visual signals.
- Autoregressive Video Generation: Supports autoregressive extension of generation length, using the last few frames of the previous video segment as motion context each time, progressively generating segment by segment to ensure temporal coherence. Introduces random perturbations, random deletions, and Classifier-free guidance strategies during training to alleviate temporal drift and error accumulation.
- Controllable Interaction Design: Keyboard actions are expressed as discrete tokens, viewpoint movement actions are expressed as continuous tokens. Based on the control module of GameFactory, it is integrated into the multimodal Diffusion Transformer architecture, using Classifier-free guidance strategies to enhance robust response capability to control signals.
Project Address of Matrix-Game
- Project Website：https://matrix-game-homepage.github.io/
- GitHub Repository：https://github.com/SkyworkAI/Matrix-Game
- HuggingFace Model Library：https://huggingface.co/Skywork/Matrix-Game
- Technical Paper：https://github.com/SkyworkAI/Matrix-Game
Application Scenarios of Matrix-Game
- Virtual Game Development: Rapidly generates diverse game maps and dynamic interactive environments, improving development efficiency and player immersion.
- Film and Television and Metaverse: Generates high-fidelity dynamic scenes, supports immersive experience development, and facilitates rapid generation of creative content.
- Embodied Intelligence Training: Provides diverse virtual environments, enhances training data for embodied intelligent agents, and improves their task execution capabilities.
- Education and Training: Create virtual teaching and vocational skills training environments to help students and trainees better understand and practice.
- Creative content generation: Provide rich materials for creative video production and virtual scene design, supporting the rapid realization of creative ideas.