What is Multiverse
Multiverse is the world's first AI-generated multiplayer game model launched by the Israeli team Enigma Labs. It is a multiplayer racing game where players can overtake, drift, and accelerate, and each action affects and reshapes the game world in real time. The model uses AI technology to generate game visuals in real time, ensuring that two players see the same logically unified world. The model is based on a diffusion model, fusing and processing players' viewpoints and actions to generate coherent and consistent game visuals. The core technology of Multiverse lies in its innovative multiplayer world model architecture, which solves the challenge of viewpoint consistency in multiplayer games through joint action vectors and dual-view channel stacking technology. The model's training cost is only $1,500 and it can run on an ordinary PC. The project code, data, weights, architecture, and research results have been fully open-sourced, providing new possibilities for the application of AI in the field of multiplayer games.
Main features of Multiverse
- Multiplayer real-time interaction: Supports real-time interaction between two players in the same virtual world, such as overtaking and collisions in a racing game, with both players' viewpoints remaining consistent.
- Dynamic world generation: Generates game visuals in real time based on players' actions and operations.
- Efficient frame prediction: Accurately predicts future game frames, ensuring the smoothness and coherence of the game.
- Low-cost operation: Supports running on an ordinary personal computer without the need for high-end hardware support, lowering the barrier to use.
Technical principles of Multiverse
- Multiplayer game architecture: To build a multiplayer game world model, the above core building blocks were retained, and the structure was broken down — the inputs and outputs were reconnected, and the training process was redesigned from scratch to achieve truly cooperative gameplay:
- Action embedder: Obtains the actions of two players and outputs an embedding representing them.
- Denoising network: A diffusion network that can simultaneously generate frames for two players as a single entity based on the previous frames and action embeddings of both players.
- Upsampler: The upsampler here receives the frames of two players separately and simultaneously computes the upsampled versions.
- Viewpoint merging solution: To create a multiplayer gaming experience, the model needs to collect the previous frames and actions of both players and output their respective predicted frames. The key is that these two outputs must not merely look good; they need to be internally consistent. Multiverse proposed a workaround: concatenate the viewpoints of the two players into a single image, fuse their inputs into a joint action vector, and treat all of this as a unified scene. Specifically, this is done by stacking along the channel axis, treating the two frame images as an image with twice the color channels. Because the diffusion model here is a U-shaped network composed mainly of convolutional layers and deconvolutional layers, the first layer only processes nearby pixels. If the two frames are stacked vertically, then the frames are not processed until the middle layer. This reduces the model's ability to produce consistent structures between frames. If the frames are stacked along the channel axis, then every layer of the network processes the views of both players simultaneously.
- Training method
- Context extension: To accurately predict the next frame, the model needs to receive the player's actions (such as steering input) and enough frames to calculate the speeds of the two cars relative to the road and to each other. Research found that 8 frames (at 30 frames per second) allow the model to learn vehicle kinematics, such as acceleration, braking, and steering. However, the relative motion speed of the two cars is much slower than that of the road. To capture this relative motion, the context size needs to be expanded by nearly three times. But doing so would make the model too slow for real-time gaming, increase memory usage, and greatly slow down training. To maintain the context size while providing more temporal information, the authors provided the model with sparse sampling of the previous frames and actions. Specifically, they provided the model with the most recent 4 frames. Then, for the next 4 frames, they provided every 4th frame. The earliest frame in the context is 20 frames back, i.e., 0.666 seconds in the past, which is enough to capture the relative motion of the vehicles. It also allows the model to better capture speed and acceleration compared to the road surface, making the driving dynamics more outstanding.
- Multiplayer Game Training: To enable the model to learn driving techniques and interactions in multiplayer games, the model needs to be trained in these interaction scenarios. Walking, driving, and other common tasks in world models usually require only short prediction horizons, such as predicting the situation 0.25 seconds into the future. Interactions in multiplayer games require longer time spans. Within a quarter of a second, the relative motion between players is almost negligible. To train a multiplayer game world model, a longer prediction horizon needs to be set. Multiverse trains the model to perform autoregressive prediction (at 30 frames per second) for up to 15 seconds into the future. To enable the model to make such long-term predictions, Multiverse adopts curriculum learning, increasing the prediction time from 0.25 seconds to 15 seconds during training. In the initial training phase, the model can be trained efficiently, when the model is learning low-level features such as cars and track geometry. The model learns to generate coherent frames and model vehicle kinematics, and is then trained on high-level concepts such as player behavior. After increasing the prediction horizon, the model's "physical consistency" and inter-frame consistency are significantly improved.
- Efficient Long-Horizon Training: Training a model for more than 100 frames into the future poses a challenge for VRAM. With larger batches, loading these frames into GPU memory for autoregressive prediction becomes infeasible. To address this memory limitation, a paging approach is used for autoregressive prediction. At the start of training, the first batch of data is loaded, and predictions are made on it. Then the next page of data is loaded, and frames beyond the context window are discarded.
Multiverse Project Address
- Project Official Website：https://enigma-labs.io/blog
- GitHub Repository：https://github.com/EnigmaLabsAI/multiverse
- HuggingFace Model Library：https://huggingface.co/Enigma-AI
Multiverse Dataset
- Data Source: The data used by the team to train the model was collected from Sony's game "Gran Turismo 4".
- Data Collection Method: Based on the in-game replay system, each race is replayed twice, recorded from each player's perspective. The two recordings are then synchronized, aligned with the original two-player race, and merged into a single video showing both players playing simultaneously. Computer vision is used to extract the throttle, brake, and steering bars displayed on the game screen frame by frame, and then the control commands are inferred. In other words, the operations can be reconstructed entirely from the visual information, without the need for additional log files.
- Automatic Data Generation: Write a script to send random inputs to the game's B-Spec mode, automatically trigger races, record replay footage from two perspectives, and capture third-person video of AI-driven races.
Multiverse Application Scenarios
- Multiplayer Game Development: Used in the development of multiplayer online games to provide a more realistic and rich interactive experience.
- VR/AR Applications: Create multi-user shared virtual environments to enhance immersion and social interaction.
- AI Training and Research: As an open-source model, train intelligent AI agents and study decision-making and collaboration in complex environments.
- Education and Training: Create virtual training scenarios for use in driving, military drills, or team collaboration training.
- Entertainment and Social Interaction: Develop applications such as virtual parties and online events to provide novel social experiences.
- Simulation Management Games: In simulation management games, players need to perform operations such as resource management, building planning, and city development. Every decision may affect the economic and ecological balance of the entire simulated world.