SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Unique3D - An Open-Source Image-to-3D Generation Model by the Tsinghua University Team

Author: AI小集 · Original source: AI工具集

What Is Unique3D

Unique3D is an open-source framework by the Tsinghua University team for singleimage-to-3D model conversion, which, by combining a multi-view diffusion model and a normal diffusion model, along with an efficient multi-stage upsampling strategy, can rapidly generate 3D meshes with high fidelity and rich textures from a single image. Unique3D further incorporates the ISOMER algorithm to ensure the consistency and accuracy of the generated 3D models in geometry and color. It takes only 30 seconds to complete the conversion from a single-view image to a 3D model, and the generation results outperform image-to-3D models such as InstantMesh, CRM, and OpenLRM.

Unique3D

Unique3D's Features

  • Single-Image 3D Mesh Generation: Unique3D can automatically generate a 3D mesh model from a single 2D image, converting a flat image into a three-dimensional form with spatial depth.
  • Multi-View Generation: The system uses a multi-view diffusion model to generate four orthographic view images of the same object. These views capture the object's features from different directions, providing comprehensive viewpoint information for 3D reconstruction.
  • Normal Map Generation: Unique3D generates a corresponding normal map for each multi-view image. These maps record the orientation information of the object's surface, which is crucial for subsequent 3D model rendering, can simulate how light interacts with the surface, and enhance the model's realism.
  • Multi-Level Resolution Enhancement: Through a multi-stage upsampling process, the resolution of the generated images is gradually increased, from low resolution to high resolution (e.g., from 256×256 to 2048×2048), making the textures and details of the 3D model clearer.
  • Integration of Geometric and Texture Details: During the reconstruction process, Unique3D tightly combines color information and geometric shape to ensure that the generated 3D model remains visually consistent with the original 2D image while possessing complex geometric structures and rich texture details.
  • High-Fidelity Output: The generated 3D model is highly consistent with the input 2D image in shape, texture, and color, meeting high-fidelity standards in both the accuracy of geometric form and the richness of texture.
Unique3D

Unique3D's Official Website Entry

Unique3D's Technical Principles

How Unique3D Works
  • Multi-View Diffusion Model: Utilizes a diffusion model to generate multi-view (usually four orthographic views) images from a single-view image. These models learn the distribution of 2D images through training and extend it to 3D space to generate images with different viewpoints.
  • Normal Diffusion Model: Works in coordination with the multi-view diffusion model to generate a corresponding normal map for each generated view image. These normal maps contain directional information of surface normals, which is crucial for subsequent 3D reconstruction.
  • Multi-Stage Upsampling Process: Adopts a multi-stage upsampling strategy to gradually increase the resolution of the generated images. The initially generated images have lower resolution, and through upsampling techniques, they are gradually enhanced to higher resolution to obtain clearer details.
  • ISOMER Mesh Reconstruction Algorithm: An efficient mesh reconstruction algorithm used to reconstruct a 3D mesh from high-resolution multi-view RGB images and normal maps. The ISOMER algorithm includes:
    • Initial mesh estimation: Rapidly generate the rough topology and initial mesh of a 3D object.
    • Coarse-to-fine mesh optimization: Through an iterative optimization process, gradually improve the shape of the mesh to bring it closer to the target shape.
    • Explicit target optimization: Specify an optimization target for each vertex, address issues caused by inconsistent viewpoints, and improve the accuracy of geometric details.
  • Integration of Color and Geometric Priors: During mesh reconstruction, integrate color information and geometric shape information into the mesh result to improve the visual realism and accuracy of the final model.
  • ExplicitTarget: Define an optimization target for each vertex, which is a mapping function from the vertex set to the color set, used to guide vertex color optimization and improve the multi-view consistency of the model.
  • Expansion Regularization: A technique used during the optimization process that avoids surface collapse by moving vertices along their normal direction, ensuring the integrity of the model.
  • Color Completion Algorithm: For color completion in invisible regions, use an efficient algorithm to smoothly propagate colors from visible regions to invisible regions, ensuring color consistency across the entire model.