What is Stable Video 3D
Stable Video 3D (SV3D for short) is a multi-view synthesis and 3D generation model developed by Stability AI. It can generate consistent multi-view images from a single image and further optimize them to generate high-quality 3D mesh models. This model improves upon the previously released Stable Video Diffusion model, providing better quality and a better multi-view experience. Compared with other 3D generation models, the main advantage of SV3D is that it uses a video diffusion model rather than an image diffusion model, which provides significant benefits in the generalization and view consistency of the generated output.
Features of Stable Video 3D
- Multi-view Video Generation: SV3D can generate video content from multiple viewpoints from a single image input. Users can view the object from different directions and angles, and each viewpoint is high-quality while maintaining consistency between viewpoints.
- 3D Mesh Creation: By using the generated multi-view videos and the Stable Video 3D model, users can create 3D meshes of objects. These 3D meshes are inferred from 2D images and can be used in various 3D applications, such as game development, virtual reality, augmented reality, and more.
- Orbital Video Generation: Stable Video 3D provides the ability to generate orbital videos around an object, allowing users to create videos that rotate or move around the object, providing a dynamic viewing experience.
- Camera Path Control: SV3D supports creating 3D videos along a specified camera path. Users can precisely control the video viewpoint and camera movement, providing greater freedom for creation.
- Novel View Synthesis (NVS): SV3D has made significant progress in novel view synthesis, capable of generating consistent and realistic views from any given angle, improving the realism and accuracy of 3D generation.
Stable Video 3D Official Website Entrance
- Official project homepage:https://sv3d.github.io/
- Technical report:https://stability.ai/s/SV3D_report.pdf
- Hugging Face model:https://huggingface.co/stabilityai/sv3d
How Stable Video 3D Works
- Novel View Synthesis (NVS):
- Input Image: A single 2D image provided by the user as input, containing one or more objects.
- Camera Pose Control: Define a camera trajectory, including a series of angles (elevation and azimuth), used to control the viewpoint of the generated images.
- Latent Video Diffusion Model: Use a well-trained latent video diffusion model (such as Stable Video Diffusion – SVD), which can generate a series of new viewpoint images based on the input image and camera pose. These images simulate an orbital video around a 3D object.
- 3D Representation Optimization:
- Coarse 3D Reconstruction: Using the generated multi-view images as targets, reconstruct a coarse representation of the 3D object by training a NeRF (Neural Radiance Fields) model. This step is performed at lower resolution to capture the approximate shape and texture of the object.
- Mesh Extraction: Extract an initial 3D mesh from the trained NeRF model, typically using the Marching Cubes algorithm.
- Fine Optimization: The DMTet (Deep Marching Tetrahedra) representation is used to further refine the 3D mesh, and this step is performed at high resolution to improve the accuracy of details and the quality of the mesh.
- Improved 3D Optimization Techniques:
- Masked Score Distillation Sampling (SDS) Loss: To improve the 3D quality of unseen regions, SV3D introduces a masked score distillation sampling loss. This loss function focuses on filling in and optimizing those regions that are not visible from the reference viewpoint during training.
- Decoupled Lighting Model: SV3D also proposes a decoupled lighting model that is optimized independently of the 3D shape and texture to reduce rendering issues caused by fixed lighting conditions.
- Training and Evaluation:
- Dataset: SV3D is trained on datasets containing diverse 3D objects, such as the Objaverse dataset.
- Evaluation: The quality of multi-view images and 3D meshes generated by SV3D is evaluated by comparison with real-world 3D data and other NVS methods.