What is GenXD
GenXD is a 3D-4D joint generation framework jointly launched by the National University of Singapore and Microsoft. It can generate high-quality 3D and 4D scenes from any number of conditional images. The framework uses a data curation pipeline to extract camera poses and object motion intensity from videos, and trains the model based on this information and the large-scale 4D dataset CamVid-30K. GenXD is based on a multi-view temporal module to decouple camera and object motion, and uses masked latent conditions to support conditional generation from multiple views, enabling multiple 3D and 4D generation tasks to be handled in a single model.
Main features of GenXD
- 3D and 4D scene generation: GenXD can generate high-quality 3D and 4D scenes from single or multiple views, including dynamic and static content.
- Camera pose estimation: Based on structure from motion (SfM) technology, GenXD estimates camera poses in videos, providing a foundation for generating videos consistent with camera trajectories.
- Object motion estimation: Based on depth estimation and keypoint tracking, GenXD identifies and simulates the motion of objects in videos.
- Multi-view temporal module: The module inside the framework processes multi-view and temporal information, decouples camera motion and object motion, and generates more realistic dynamic scenes.
- Masked latent conditions: GenXD supports conditional generation with masked latent conditions, allowing the model to accept any number of input views without changing the network architecture.
Technical principles of GenXD
- Data curation pipeline: GenXD is based on a data curation pipeline to extract camera pose and object motion information from videos, providing the necessary data for subsequent model training.
- Multi-view temporal module: The multi-view temporal module inside GenXD can process multi-view and temporal information, using an alpha fusion strategy for seamless learning in 3D and 4D data.
- Masked latent conditional diffusion model: GenXD uses a masked latent conditional diffusion model (LDM) to generate images at different camera viewpoints and time steps, supporting single-view and multi-view generation.
- Decoupling camera and object motion: Based on the multi-view temporal module, GenXD separates camera motion and object motion, which is crucial for generating dynamic scenes.
- 3D and 4D data fusion: GenXD combines 3D and 4D data during training, allowing the model to learn spatial and temporal information simultaneously and improve generation quality.
- 3D representation optimization: The images generated by GenXD are directly used to optimize 3D representations, such as 3D Gaussian splatting (3D-GS) and Zip-NeRF, achieving high-quality 3D scene reconstruction.
GenXD project address
- Project website：gen-x-d.github.io
- GitHub repository：https://github.com/HeliosZhao/GenXD
- arXiv technical paper：https://arxiv.org/pdf/2411.02319
Application scenarios of GenXD
- Video game development: GenXD is used to generate 3D and 4D environments in games, providing more realistic and dynamic game worlds.
- Film and visual effects: In film production, GenXD creates complex 3D scenes and special effects, reducing the cost of actual filming and post-production.
- Virtual reality (VR) and augmented reality (AR): GenXD generates immersive 3D and 4D content, enhancing the user experience of VR and AR applications.
- Architecture and urban planning:Based on 3D models generated by GenXD, architects and urban planners can present design concepts and planning schemes more intuitively.
- Education and training: GenXD creates simulated environments for use in education and professional training, such as simulated surgery, historical reenactments, and so on.