What is ClotheDreamer
ClotheDreamer is a 3D clothing generation technology jointly launched by Shanghai University, Shanghai Jiao Tong University, Fudan University, and Tencent Youtu Lab. It can generate high-fidelity, wearable 3D clothing assets from text descriptions. ClotheDreamer uses 3D Gaussians as its foundation and, based on Disentangled Clothe Gaussian Splatting (DCGS), achieves separate optimization of clothing and the human body, while using bidirectional Score Distillation Sampling (SDS) to improve clothing rendering quality. ClotheDreamer supports custom clothing templates, allowing the generated clothing to fit different body types, and is suitable for virtual try-on and physically accurate animation.
Main Features of ClotheDreamer
- Text-driven 3D clothing generation: Automatically generates the corresponding 3D clothing model based on text descriptions.
- High-fidelity rendering: The generated 3D clothing has a high level of detail and realism.
- Wearability: The generated clothing models are wearable and can be used for virtual try-on.
- Physically accurate animation: Supports physically accurate animation effects for the generated clothing.
- Custom template input: Supports users uploading custom clothing templates to generate personalized 3D clothing.
- Adaptation to different body types: The generated clothing can fit virtual characters of different body types.
Technical Principles of ClotheDreamer
- Disentangled Clothe Gaussian Splatting (DCGS): A novel representation method that represents clothing and the human body separately as Gaussian models, supporting independent optimization and rendering.
- Bidirectional Score Distillation Sampling (SDS): Uses a pretrained 2D diffusion model to optimize 3D clothing and human body rendering, guiding the RGBD rendering of the clothing and the human body separately to improve generation quality.
- Text description parsing: Uses a language model (such as ChatGPT) to parse text descriptions, determine the type and features of the clothing, and provide a basis for initializing the clothing model.
- Zero-shot learning: Generates the corresponding 3D clothing model without having seen the specific type of clothing.
- New pruning strategy: A pruning strategy proposed for loose clothing that avoids mistakenly removing useful Gaussian points during optimization, preserving the integrity of the clothing.
- Template-guided clothing generation: Uses custom clothing template meshes to guide the generation of 3D clothing, improving the personalization and practicality of the generation process.
ClotheDreamer Project Address
- Project official website：ggxxii.github.io/clothedreamer
- GitHub repository：https://github.com/ggxxii/clothedreamer
- arXiv technical paper：https://arxiv.org/pdf/2406.16815
Application Scenarios of ClotheDreamer
- Fashion design: Designers can quickly generate and iterate 3D clothing models, accelerating the design process and previewing design effects in a virtual environment.
- Virtual try-on: In e-commerce, customers can try on clothing in a virtual environment, improving the shopping experience and reducing return rates.
- Games and entertainment: Game developers can design diverse clothing for game characters, providing richer and more personalized character customization options.
- Film and animation production: In the film and animation industry, 3D clothing can be quickly generated and animated, improving production efficiency and reducing costs.
- Virtual reality (VR) and augmented reality (AR): In VR and AR applications, provide users with an immersive clothing try-on and design experience.