What is FabricDiffusion
FabricDiffusion is a high-fidelity 3D garment generation technology jointly launched by Google and Carnegie Mellon University. It can transfer the textures and prints of 2D garment images from the real world to 3D garment models of arbitrary shapes with high quality. Based on denoising diffusion models and a large-scale synthetic dataset, FabricDiffusion corrects the distortion of input texture images and generates various texture maps, including diffuse, roughness, normal, and metallic maps, achieving accurate relighting and rendering of 3D garments under different lighting conditions, demonstrating excellent performance and generalization ability.
Main Features of FabricDiffusion
- High-Quality Texture Transfer: Automatically extracts and transfers the textures and prints of 2D garment images to 3D garment models.
- Handles Multiple Textures: Handles various types of textures, patterns, and materials.
- Generates Multiple Texture Maps: Can generate diffuse maps, as well as roughness, normal, and metallic maps.
- Rendering Across Lighting Conditions: Supports accurate relighting and rendering of 3D garments under different lighting conditions.
- Zero-Shot Generalization: Generalizes to real-world images despite being trained entirely on synthetic rendered images.
Technical Principles of FabricDiffusion
- Denoising Diffusion Model: Uses a denoising diffusion model to learn to recover undistorted, tileable texture materials from distorted input texture images.
- Large-Scale Synthetic Dataset: Constructs a large-scale synthetic dataset containing over 100k textile color images, 3.8k material PBR texture maps, 7k prints, and 22 3D garment meshes to train the model.
- Texture Image Correction: Based on model training, corrects distortion in input texture images and generates flat texture maps that are closely integrated with physically based rendering (PBR) material generation pipelines.
- Feature Transfer: Transfers various features from a single garment image, including texture patterns, material properties, and detailed prints and logos.
- Normalization and Tileability: The generated texture maps are normalized and tiled in the garment's UV space, seamlessly integrating with existing PBR material estimation pipelines.
- Conditional Generation: The model generates corresponding textures conditioned on the input garment image, achieving high-quality texture transfer from 2D to 3D.
Project Address of FabricDiffusion
- Project Website：humansensinglab.github.io/fabric-diffusion
- arXiv Technical Paper：https://arxiv.org/pdf/2410.01801
Application Scenarios of FabricDiffusion
- Virtual Try-On: In e-commerce and fashion retail, creates virtual fitting rooms that allow consumers to try on 3D garments online, improving the shopping experience.
- Games and Entertainment: In game development, quickly generates 3D garments with realistic textures, enhancing the visual realism of game characters.
- Virtual Reality (VR) and Augmented Reality (AR): In VR and AR applications, creates realistic virtual environments and characters, enhancing user immersion.
- Film and Television Production: In film and television production, generates or modifies garment textures, improving the efficiency of visual effects and costume design.
- Fashion Design and Prototyping: Designers explore new designs and textures, quickly create clothing prototypes, and accelerate design iteration.