VideoWorld is a deep generative model that explores "learning knowledge from unlabeled raw videos." Unlike current mainstream large language models (LLMs), which mainly learn from text, VideoWorld attempts to acquire complex knowledge solely through visual input, including task rules, reasoning, and planning abilities. The research team built two experimental environments, video Go and robotic manipulation, collected purely visual training data, and adopted an architecture of VQ-VAE plus an autoregressive Transformer, trained under the "next token (or next frame) prediction" paradigm. At test time, the model generates new frames based on previous frames and derives task actions such as Go moves or robotic manipulation from them.
Main capabilities and key findings:
- Video-only training can provide enough information to learn knowledge, including rules, reasoning, and planning abilities.
- Improving the compactness of visual representations can significantly improve learning efficiency. To this end, the team proposed the latent dynamics model (LDM) as a key component of VideoWorld, compressing future visual changes into a set of latent codes as a compact representation of multi-step visual context. During training, the model simultaneously predicts video frames and latent codes, thereby better capturing and reasoning about diverse visual information such as object interactions and scene dynamics.
In the Go task, VideoWorld, with only a 300-million-parameter model, reached a 5-dan professional level on Video-GoBench, without relying on search algorithms or reward mechanisms in reinforcement learning. In the robotic task, VideoWorld effectively learned diverse control manipulations and generalized across environments, approaching oracle model performance on CALVIN and RLBench.
Usage: The project page provides open-source code, data, and models for further research. Users can visit https://maverickren.github.io/VideoWorld.github.io/ for detailed information. Specific installation commands, dependencies, and running steps are not currently provided on the project page and need to be checked in the open-source repository.
Applicable scenarios:
- Studying the knowledge learning ability of video generation models.
- Learning task rules, reasoning, and planning from purely visual data.
- Video Go playing (Video-GoBench benchmark).
- Robotic manipulation control and cross-environment generalization.
- Serving as a benchmark and tool for exploring visual knowledge acquisition.
Limitations:
- The current experiments are limited to two environments: Go and robotic manipulation.
- Model performance is affected by the compactness of visual representations; redundant representations reduce learning efficiency.
- Detailed installation, usage steps, and parameter configuration are not yet explained in the provided materials.
