What is GEN3C
GEN3C is a new generative video model launched by NVIDIA, the University of Toronto, and the Vector Institute, generating high-quality 3D video content based on precise camera control and spatiotemporal consistency. GEN3C builds on point cloud-based 3D cache guidanceVideo Generation, using depth estimation of input images or video frames to back-project and generate a 3D scene, and rendering 2D video according to a user-provided camera trajectory as conditional input for the generative model. The core advantage of GEN3C lies in precise control of camera motion, avoiding the inconsistency problems caused by the lack of explicit 3D modeling in traditional methods. GEN3C supports video generation from single-view to multi-view, is suitable for static and dynamic scenes, and has achieved state-of-the-art results in novel view synthesis tasks from sparse views. GEN3C supports 3D editing and the generation of complex camera movements (such as dolly zoom), providing a powerful tool for video creation and simulation.
Main Features of GEN3C
- Precise camera control: Generates video according to a user-specified camera trajectory, supports complex camera movements (such as dolly zoom, rotating shots, etc.), and maintains the spatiotemporal consistency of the video.
- 3D-consistent video generation: Supports generating realistic and consistent videos, avoiding problems such as objects suddenly appearing or disappearing.
- Novel view synthesis from multi-view and sparse views: Supports input from single-view, sparse multi-view, to dense multi-view, generating high-quality novel view videos.
- 3D editing and scene manipulation: Users modify the 3D point cloud (such as deleting or adding objects) to edit the scene and generate the corresponding video.
- Long video generation: Supports generating long videos while maintaining spatiotemporal consistency.
Technical Principles of GEN3C
- Building a 3D cache: The depth estimation of the input image or video frame is back-projected to generate a 3D point cloud, forming a spatiotemporally consistent 3D cache. The cache serves as the foundation for video generation and provides an explicit 3D structure of the scene.
- Rendering the 3D cache: According to the user-provided camera trajectory, the 3D cache is rendered into a 2D video.
- Video generation: Using a pretrained video diffusion model (such as Stable Video Diffusion or Cosmos), the rendered 3D cache is used as conditional input to generate high-quality video. Based on the denoising objective in the optimized diffusion process, the model repairs artifacts in the rendering and fills in missing information.
- Multi-view fusion: When the input contains multiple views, GEN3C uses a max-pooling-based fusion strategy to aggregate information from different views into the video generation model, generating consistent video.
- Autoregressive generation and cache updating: For long video generation, GEN3C divides the video into multiple overlapping chunks, generates them chunk by chunk, and updates the 3D cache to maintain the spatiotemporal consistency of the video.
GEN3C Project Address
- Project Website：https://research.nvidia.com/labs/toronto-ai/GEN3C/
- GitHub Repository：https://github.com/nv-tlabs/GEN3C
- arXiv Technical Paper：https://arxiv.org/pdf/2503.03751
Application Scenarios of GEN3C
- Single-view video generation: Generates dynamic video from a single image, suitable for rapid content creation.
- Novel view synthesis: Generates novel view video from a small number of views, used in VR/AR and 3D reconstruction.
- Driving simulation: Generate videos from different viewpoints of driving scenes to aid autonomous driving training.
- Dynamic video re-rendering: Generate new viewpoints for existing videos, used in video editing and secondary creation.
- 3D scene editing: Generate new videos after modifying scene content, aiding film and television production and game development.