What is GeneMAN
GeneMAN is a 3D human creation framework jointly launched by Shanghai AI Laboratory, Peking University, Nanyang Technological University, and Shanghai Jiao Tong University. It can create high-fidelity 3D human models from a single image. The framework does not rely on parametric human models; instead, it uses multi-source high-quality human datasets, including 3D scans, multi-view videos, single photos, and synthetic data, to train human-specific 2D and 3D prior models. Based on geometric initialization, a sculpting process, and multi-space texture refinement technology, GeneMAN achieves the goal of extracting high-quality 3D human models from images of natural environment data, regardless of variations in human proportions, poses, or clothing in the images.
Main Features of GeneMAN
- Handles diverse human proportions: Whether the person in the image is full-body, half-body, or in close-up, GeneMAN can reconstruct the corresponding 3D model.
- Adapts to different clothing and poses: GeneMAN can handle various clothing and human poses, including natural poses and common objects.
- Handles in-the-wild images: GeneMAN is particularly good at handling images in natural environments, the so-called "in-the-wild data," which may contain various complex backgrounds and lighting conditions.
- Generates high-quality textures: Based on a multi-space texture refinement process, GeneMAN can generate 3D human textures that are rich in detail and consistent with the input image.
Technical Principles of GeneMAN
- 2D and 3D human prior model training：
- Text-to-image diffusion model: Train a human-specific text-to-image diffusion model for generating 2D human priors.
- View-conditioned diffusion model: Train a view-conditioned diffusion model for generating 3D human priors.
- Geometric initialization and sculpting process：
- Template-free geometric initialization: Use NeRF (Neural Radiance Field) technology to initialize a template-free 3D geometric shape.
- Combining priors and losses: Combine GeneMAN's 2D and 3D priors and structured difference loss (SDS loss) to guide geometric initialization.
- Reference loss: Use reference loss to ensure alignment with the input image.
- High-resolution refinement: Convert NeRF to DMTet (Deep Marching Tetrahedra) for high-resolution geometric refinement, guided by pretrained human-specific normal and depth adaptation diffusion models.
- Multi-space texture refinement process：
- Coarse texture generation: Use multi-view texture technology to generate coarse textures.
- Latent space refinement: Iteratively refine the texture in latent space.
- Pixel space refinement: Obtain detailed textures in pixel space based on an optimized UV map, optimized using ControlNet based on 2D priors.
GeneMAN Project Address
- Project official website：roooooz.github.io/GeneMAN
- GitHub repository：https://github.com/roooooz/GeneMAN(coming soon open source)
- arXiv technical paper：https://arxiv.org/pdf/2411.18624
Application Scenarios of GeneMAN
- Virtual try-on: Users upload their full-body photos to generate 3D models for online clothing try-on, providing a more realistic fitting experience.
- Games and entertainment: In game and film production, create or modify character models to provide more personalized and realistic character designs.
- Augmented Reality (AR) and Virtual Reality (VR): In AR and VR applications, generate users' 3D avatars to improve immersion and interactivity.
- Fashion Design: Designers quickly convert design sketches into 3D models, accelerating the design process.
- Fitness and Health Management: Based on analyzing users' 3D models, provide personalized fitness guidance and health advice.