What is Hi3DGen
Hi3DGen is a high-fidelity 3D geometry generation framework jointly launched by researchers from The Chinese University of Hong Kong (Shenzhen), ByteDance, and Tsinghua University. It can generate high-fidelity3D modelsfrom 2D images. By using normal maps as an intermediate representation, Hi3DGen can generate rich geometric details and significantly outperforms existing methods. The framework contains three key components: an image-to-normal estimator, a normal-to-geometry learning method, and a 3D data synthesis pipeline.
Main Features of Hi3DGen
- Generate high-fidelity 3D models from 2D images: It can convert 2D images into 3D geometric models with rich details.
- Image-to-normal estimation: Through noise injection and dual-stream training, low-frequency and high-frequency image patterns are decoupled, achieving generalizable, stable, and sharp normal estimation.
- Normal-to-geometry learning: Latent diffusion learning based on normal regularization enhances the fidelity of 3D geometry generation.
- 3D data synthesis: Build high-quality 3D datasets to support training.
Technical Principles of Hi3DGen
- Image-to-normal estimator: The component decouples the low-frequency and high-frequency patterns of images through noise injection and dual-stream training. The low-frequency patterns are responsible for the overall shape and structure, while the high-frequency patterns are responsible for details and textures. It can generate generalizable, stable, and sharp normal maps, providing a high-quality intermediate representation for subsequent 3D geometry generation.
- Normal-to-geometry learning method: Based on normal maps as a regularization means, the latent diffusion model is trained. It enhances the fidelity of 3D geometry generation, allowing the generated 3D models to retain more details.
- 3D data synthesis pipeline: Through the 3D data synthesis pipeline, high-quality 3D datasets are constructed for training models. It supports the model in learning the mapping relationship from 2D images to 3D geometry.
- Two-stage generation process: Hi3DGen adopts a two-stage generation process:
- Stage 1: Basic multi-view generation: Using a pre-trained video diffusion model, fine-tuned with additional camera pose conditions, it converts a single-view image into a low-resolution 3D-aware sequence of images (orbit video).
- Stage 2: 3D-aware multi-view refinement: The low-resolution multi-view images generated in the first stage are input into a 3D-aware video-to-video refiner to further improve image resolution and texture details.
- 3D Gaussian Splatting (3DGS): Learn an implicit 3D model from the generated high-resolution multi-view images, and render additional interpolated views through 3DGS.
- SDF-based reconstruction: Through an SDF (Signed Distance Function)-based reconstruction method, high-quality 3D meshes are extracted from the enhanced dense views.
Project Address of Hi3DGen
- Project official website：https://stable-x.github.io/Hi3DGen/
- Github repository：https://github.com/Stable-X/Hi3DGen
Application Scenarios of Hi3DGen
- Game development: Quickly generate high-quality 3D game assets, such as characters, props, and scenes.
- Film and television production: Used to create realistic 3D special effects and animations, saving the time and cost of traditional modeling.
- 3D visualization: View and analyze 3D models from different angles, suitable for fields such as architectural design and industrial design.
- Virtual photography: Generate high-quality images from different viewpoints for online display and marketing.
- Cultural relic preservation: Reconstruct 3D models from a single photograph of a cultural relic for digital preservation and research.
- Medical imaging: Generate 3D models from medical images (such as X-rays and CT scans) to assist in diagnosis and treatment.