What Is VerseCrafter
VerseCrafter is a dynamic real-world video world model launched by Fudan University together with Tencent PCG ARC Lab and other institutions, featuring 4D geometric control capabilities. The model is trained on the large-scale real-world dataset VerseControl4D and can handle complex dynamic scenes while maintaining strong spatiotemporal consistency. Users can specify camera trajectories and object trajectories to generate high-quality, geometrically consistent videos. The model has broad application prospects in fields such as video generation, virtual reality, and game development.
Main Features of VerseCrafter
-
4D Geometric Control: Users can precisely control the viewpoint and object motion in videos by specifying camera trajectories and multi-object 3D Gaussian trajectories.
-
Flexible Control Modes: The model supports independent camera control, independent object control, and joint control of camera and objects, meeting the needs of different scenarios.
-
High-Quality Video Generation: While maintaining the realism of videos, it ensures the geometric consistency of generated videos and avoids distortion.
-
Multi-View Consistency: The model can generate consistent video content from different viewpoints, making it suitable for multi-person interaction scenarios.
-
Large-Scale Data Support: Trained on the VerseControl4D dataset, covering both dynamic and static scenes, it improves the model's generalization ability.
Technical Principles of VerseCrafter
- Frozen Wan2.1 Backbone Network: The model adopts the pretrainedWan2.1as the base model, maintaining powerful video generation and generalization capabilities while injecting geometric control signals on top of Wan2.1.
- GeoAdapter: A lightweight geometric adapter encodes 4D control signals (camera trajectories and 3D Gaussian trajectories) into multi-channel maps and injects them into Wan2.1's diffusion blocks to achieve precise control.
- 4D Control Signal Rendering: Camera trajectories and object trajectories are rendered into background RGB/depth maps and 3D Gaussian trajectory maps, which are input to the generative model as conditional signals.
- VerseControl4D Dataset: Through large-scale real-world video data, camera trajectories and object trajectories are extracted, providing rich geometric supervision for model training and supporting the generation of dynamic and static scenes.
Project Addresses of VerseCrafter
- Project Official Website：https://sixiaozheng.github.io/VerseCrafter_page/
- GitHub Repository：https://github.com/TencentARC/VerseCrafter
- HuggingFace Model Library：https://huggingface.co/TencentARC/VerseCrafter
- arXiv Technical Paper：https://arxiv.org/pdf/2601.05138
Application Scenarios of VerseCrafter
- Virtual Reality (VR) and Augmented Reality (AR): VerseCrafter can build immersive virtual worlds, supporting users in exploring scenes in real time through camera and object motion control, enhancing the interactive experience.
- Game Development: It generates dynamic backgrounds and object motion for games, optimizes viewpoint switching and rendering effects, and reduces development costs.
- Video Content Creation: Creators can use VerseCrafter to quickly generate high-quality dynamic videos, meeting creative needs in advertising, film, and animation production.
- Education and Training: VerseCrafter can create realistic virtual teaching scenarios, such as historical reenactments and scientific experiment simulations, enhancing students' interest and participation in learning.
- Entertainment and Media: It is used to develop interactive video content, such as choose-your-own-adventure videos, where viewers can change the direction of the story by controlling viewpoints and object motion.