What is Fashion-VDM
Fashion-VDM is a virtual try-on technology based on video diffusion models (VDM), jointly launched by Google and the University of Washington. Given a garment image and a person video, it can generate a high-quality try-on video of the person wearing the specified garment, preserving the person's identity and motion. Fashion-VDM is based on a diffusion model architecture, split classifier-free guidance, and a progressive temporal training strategy, addressing the issues of garment detail and temporal consistency in video virtual try-on. When video data is limited, joint image-video training improves garment fidelity, reaching a new level in the field of video virtual try-on.
Main Features of Fashion-VDM
- Generate virtual try-on videos: Fashion-VDM combines a given garment image and person video to generate a video of the person wearing that garment, preserving the person's original identity and motion.
- Maintain garment details and temporal consistency: Addresses the issues of missing garment details and insufficient temporal consistency in existing video virtual try-on methods.
- Improve garment fidelity: Based on split classifier-free guidance technology, it enhances control over garment image details and fidelity.
- Optimize video generation efficiency: Uses a progressive temporal training strategy to achieve single-pass 512px video generation of 64 frames, improving the efficiency and quality of video generation.
- Joint image-video training: Combines image and video data during training, especially when video data is limited, to improve model performance.
Technical Principles of Fashion-VDM
- Diffusion model architecture: Fashion-VDM is based on a diffusion model, progressively recovering clear images or videos from noise.
- 3D convolutions and temporal attention blocks: Adds 3D convolutions and temporal attention blocks to the main UNet network to maintain temporal consistency between video frames.
- Split Classifier-Free Guidance: A technique for independently controlling multiple conditional signals, enabling the model to more precisely control generation results and improve garment fidelity and consistency between video frames.
- Progressive temporal training: The model is trained in multiple stages, gradually increasing video frame length, starting from image data and progressively introducing longer video frames for training until reaching the target of 64 frames.
- Joint image-video training: During training, uses image and video data with conditional network branching technology, skipping updates to temporal blocks during image batch training, improving data diversity and training stability.
- Preprocessing and encoding: Input videos and garment images are preprocessed to separately extract information such as person pose, garment-agnostic frames, garment segmentation, and garment pose, which are then encoded using different UNet encoders.
Fashion-VDM Project Address
- Project official website：johannakarras.github.io/Fashion-VDM
- arXiv technical paper：https://arxiv.org/pdf/2411.00225
Application Scenarios of Fashion-VDM
- Online clothing shopping: Before purchasing, consumers can use the virtual try-on feature to more intuitively see how the garment looks when worn, reducing return and exchange issues caused by unsuitable sizes or styles.
- Fashion design and marketing: Fashion designers and brands showcase their clothing works, predict market reactions in advance, and reduce design and marketing costs.
- E-commerce platforms: E-commerce platforms can integrate Fashion-VDM technology to enhance the interactivity and fun of online shopping and increase consumers' willingness to purchase.
- Virtual fashion shows: Showcase clothing in virtual fashion shows, providing more exciting and realistic display effects.
- Social media marketing: Brands and influencers can create engaging content on social media to increase user participation and brand exposure.