What is Solaris
Solaris is the first multi-player video world generation model, capable of simultaneously generating consistent first-person perspectives for two players in Minecraft. The model breaks through the limitation of existing models supporting only a single player, ensuring spatial consistency across players' perspectives—when one player builds or moves, the other perspective reflects the changes synchronously. The team developed the SolarisEngine data system in-house, collected 12.6 million frames of multi-player gameplay data, and innovatively introduced the Checkpointed Self Forcing training method to solve the long-sequence memory bottleneck.
Solaris's main features
- Synchronized multi-player perspective generation: Solaris can simultaneously generate consistent first-person videos for two players, ensuring spatial consistency across players' perspectives; when one player performs an action, the other player's perspective reflects the changes in real time.
- Stable long-duration generation: Through the Checkpointed Self Forcing technique, Solaris can generate stable video sequences up to 224 frames (11.2 seconds), effectively avoiding visual degradation caused by error accumulation.
- Action-conditioned control: The model accepts complete Minecraft action inputs (including movement, camera, mining, placing, etc.), and the generated videos strictly follow the given action sequences.
- Complex dynamic simulation: Solaris can simulate complex game dynamics such as inventory state synchronization, weather changes, physical building and destruction, and PvP combat.
Solaris's technical principles
- Multi-player DiT architecture: Based on MatrixGame 2.0's single-player diffusion Transformer, it supports complete Minecraft inputs by extending the action space, introduces cross-player self-attention layers to enable information exchange between two players, and adds player ID embeddings to distinguish different perspectives, while the remaining modules (cross-attention, FFN) remain unchanged from the single-player setup.
- Four-stage progressive training: Starting from single-player pretrained weights, fine-tuning on the VPT dataset to adapt to the Minecraft action space, then switching to multi-player data to train a bidirectional model as the teacher, causalizing it into a sliding-window generator, and achieving stable long-sequence generation through Checkpointed Self Forcing.
- Checkpointed Self Forcing: To solve the memory bottleneck of sliding-window autoregression, this method first generates without gradients and caches clean frames and noise states, then recomputes in a single parallel pass through a custom attention mask, strictly reproducing the sliding-window dependencies, reducing memory fromO(Lt⋅Ls) toO(Lt) , while supporting KV cache gradient backpropagation to improve generation quality.
- SolarisEngine data system: To address the lack of multi-player support in existing frameworks, the team built a camera-separated architecture based on a Mineflayer controller and the official Minecraft client, synchronizing states in real time through server plugins, using Docker containerization to achieve parallel scaling and automatic fault recovery, and ultimately collecting 12.6 million frames of action-annotated multi-player gameplay data.
Solaris's project addresses
- Project website：https://solaris-wm.github.io/
- GitHub repository：https://github.com/solaris-wm/solaris
- HuggingFace model library：https://huggingface.co/collections/nyu-visionx/solaris-models
- arXiv technical paper：https://arxiv.org/pdf/2602.22208
Solaris's application scenarios
- Embodied intelligence training and evaluation: As a multi-agent world simulator, it provides synthetic training data for robots and game AI, supports policy learning, inference-time planning, and safety evaluation, avoiding the high cost of trial and error in real environments.
- Multi-agent collaboration research: Simulating multi-agent collaborative tasks (such as joint construction and team combat), used to train AI agents' collaboration and communication abilities, and to study emergent behavior and social intelligence.
- Vision-Language-Action Model Development: The model can generate large-scale multi-view video-action-language aligned data, supporting the pre-training and fine-tuning of VLA models, and compensating for the scarcity of real human multi-agent interaction data.
- 3D Scene Understanding and Spatial Reasoning Benchmark: As a controllable testing platform, it evaluates model performance on core 3D understanding capabilities such as viewpoint consistency, object permanence, and spatial memory.