What is MineWorld
MineWorld is an open-source real-time interactive world model based on Minecraft from Microsoft Research. Built on a vision-action autoregressive Transformer architecture, it converts game scenes and actions into discrete token IDs and is trained through next-token prediction. The model features a parallel decoding algorithm that achieves a generation speed of 4 to 7 frames per second, enabling real-time interaction. MineWorld outperforms existing models such as Oasis in video quality, controllability, and inference speed.
MineWorld's Main Features
- High Generation Quality: Based on a vision-action autoregressive Transformer, MineWorld can generate coherent, high-fidelity game frames based on vision and actions.
- Strong Controllability: The model demonstrates precise and consistent behavior through action-following capability benchmarks, and can generate accurate game scenes based on input actions.
- Fast Inference Speed: By adopting a parallel decoding algorithm, the model can generate images at a speed of 4 to 7 frames per second, supporting real-time interaction.
- As a Game Agent: MineWorld simultaneously predicts game states and actions during training, and can autonomously play the game as an independent game agent.
- Real-Time Interaction Capability: Users can interact with the model in real time through a Web demo or by running it locally, selecting the initial frame, controlling camera movement, and executing game actions.
MineWorld's Technical Principles
- Vision-Action Autoregressive Transformer: MineWorld achieves joint modeling of vision and actions by converting game scenes and player actions into discrete token sequences. Specifically:
- Visual Tokenizer: It adopts a VQ-VAE architecture to segment game scenes into discrete visual tokens. The tokenizer starts from a pretrained checkpoint and is fine-tuned on the Minecraft dataset to achieve high-quality image reconstruction.
- Action Tokenizer: It quantizes the player's continuous actions (such as mouse movement) into discrete tokens, and categorizes discrete actions (such as moving forward and attacking) into different categories, each represented by a unique token.
- Transformer Decoder: It adopts the LLaMA architecture, takes interleaved sequences of visual tokens and action tokens as input, and is trained through next-token prediction. The decoder can simultaneously learn rich representations of game states as well as the conditional relationships between states and actions.
- Parallel Decoding Algorithm: To achieve real-time interaction, MineWorld has developed a parallel decoding algorithm. The algorithm leverages the spatial dependencies between adjacent image tokens to simultaneously predict spatially redundant tokens in each frame. Compared with traditional autoregressive decoding algorithms, it can significantly improve generation speed, enabling the model to achieve a generation speed of 4 to 7 frames per second at different scales.
- Training: The model is trained through next-token prediction to learn the dynamic evolution patterns between game states and the associations between actions and states.
- Inference: During the inference stage, the model generates subsequent game scenes based on the input current game state and actions. The application of the parallel decoding algorithm enables the model to quickly generate high-quality game frames.
- Evaluation Metrics: MineWorld proposes new evaluation metrics to assess the visual quality of generated scenes and action-following capability. For example, by comparing the accuracy between the predicted actions in the generated scenes and the input ground-truth actions, the model's controllability is quantified.
MineWorld's project address
- Github repository：https://github.com/microsoft/MineWorld
- HuggingFace model library：https://huggingface.co/microsoft/mineworld
- arXiv technical paper：https://arxiv.org/pdf/2504.08388
Application scenarios of MineWorld
- Embodied intelligence research: MineWorld provides a high-fidelity, interactive virtual environment that can simulate complex physical rules and dynamic scenes, making it very suitable for embodied intelligence research. Researchers can use the model to train agents to learn how to perform tasks in the virtual environment, such as object localization and navigation, environment exploration, etc.
- Reinforcement learning training: MineWorld's real-time interaction capability and high generation quality make it an ideal platform for reinforcement learning training. Researchers can use the model to quickly generate large amounts of training data, helping agents learn optimal policies in simulated environments.
- Game agent development: Because MineWorld simultaneously predicts game states and actions during training, it has the potential to serve as a game agent. Given an initial game state and action, the model can iteratively generate future states and actions, simulating long-term gameplay.
- Real-time interactive simulation: MineWorld's fast inference speed (4 to 7 frames per second) can support real-time interaction with game players.
- Video generation and editing: MineWorld can generate high-quality, coherent game videos. It can be used for video content creation, such as generating game trailers, instructional videos, etc.