What Is FitDiT
FitDiT is a high-fidelity virtual try-on technology jointly launched by Tencent and Fudan University. Based on Diffusion Transformers (DiT), it focuses on high-resolution features to improve the presentation of garment details. FitDiT uses a garment texture extractor and garment prior evolution technology to enhance its ability to capture garment textures such as stripes, patterns, and text. It uses a dilation-relaxation mask strategy to optimize garment size fitting. FitDiT performs excellently in qualitative and quantitative evaluations, can quickly generate try-on images with realism and complex details, and has fast inference speed, bringing a breakthrough to the field of virtual try-on.
Main Features of FitDiT
- High-Fidelity Virtual Try-On: Generates realistic try-on images, allowing users to see the effect of wearing specific garments in different scenarios.
- Texture-Aware Preservation: Based on a garment texture extractor and garment prior evolution, it accurately captures and reproduces complex textures on garments, such as stripes, patterns, and text.
- Size-Aware Fitting: Uses a dilation-relaxation mask strategy to adapt to the length and shape of different garments, prevent leakage of garment shape information during cross-category try-on, and achieve more accurate garment fitting.
- Fast Inference: While maintaining high-fidelity try-on results, it optimizes the DiT structure so that the inference time for a single 1024×768 image is only 4.57 seconds, improving the efficiency of the try-on process.
Technical Principles of FitDiT
- Diffusion Transformers (DiT): FitDiT is based on the DiT architecture, allocating more parameters and attention to high-resolution features to enhance the processing of garment details.
- Garment Texture Extractor: Introduces a dedicated garment texture extractor and fine-tunes garment features based on garment prior evolution to better capture rich garment details.
- Frequency-Domain Learning: Based on a customized frequency distance loss function, it enhances high-frequency garment details and improves the fidelity of garment textures and details.
- Dilation-Relaxation Mask Strategy: To solve the problem of size-aware fitting, it adopts a dilation-relaxation mask strategy to adapt to the correct length of garments, prevent generating garments that cover the entire mask area during cross-category try-on, and improve try-on accuracy.
- Structural Slimming: Optimizes the DiT structure, removes the text encoder that has little impact on virtual try-on, reduces the number of model parameters, and improves the speed of model training and inference.
- Hybrid Attention Mechanism: Uses a hybrid attention mechanism in DenoisingDiT to inject garment features extracted from GarmentDiT into the denoising process, achieving the fusion of high-resolution features.
FitDiT Project Address
- Project Website：byjiang.com/FitDiT
- GitHub Repository：https://github.com/BoyuanJiang/FitDiT
- arXiv Technical Paper：https://arxiv.org/pdf/2411.10499
Application Scenarios of FitDiT
- E-Commerce Platforms: Clothing retail websites allow consumers to see the effect of wearing different garments when shopping online, improving the shopping experience and satisfaction.
- Fashion Industry: Designers showcase their designs, allowing customers to preview the actual wearing effect of garments before purchase, increasing the appeal of the designs.
- Personalized Customization: Clothing customization services provide customers with a personalized try-on experience, ensuring that the size and style of customized clothing fully meet the customer's needs.
- Augmented Reality (AR) and Virtual Reality (VR): In AR and VR applications, provide a more realistic try-on experience, allowing users to try on clothing in a virtual environment and dress up virtual avatars.
- Social Media: Social media platforms, enabling users to try on different clothing styles when sharing photos or videos, increasing interactivity and entertainment.