SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

DreamVVT - A Video Virtual Try-On Technology Jointly Launched by ByteDance and Tsinghua University

Author: AI小集 · Original source: AI工具集

What is DreamVVT

DreamVVT is a Video Virtual Try-On (VVT) technology jointly launched by ByteDance and Tsinghua University (Shenzhen). Based on the Diffusion Transformers (DiTs) framework, it achieves high-fidelity and temporally coherent virtual try-on results through a two-stage method. In the first stage, key frames are sampled from the input video, and a Vision Language Model (VLM) is combined to generate semantically consistent try-on images; in the second stage, skeleton maps and motion information are used, combined with a pre-trained video generation model, to ensure the dynamic coherence of the video. DreamVVT can preserve clothing details under complex actions and scenes, supports try-on of complete outfits, and can dress cartoon characters in real clothing.

DreamVVT

Main Features of DreamVVT

  • High-Fidelity Virtual Try-On: It can achieve high-fidelity clothing try-on effects in videos, preserving clothing details and textures, and maintaining high-quality visual effects even under complex actions and scenes.
  • Temporal Coherence: Through a two-stage method, it ensures the temporal coherence of video try-on results, avoiding jumps or unnatural transitions, making the entire try-on process look smooth and natural.
  • Multi-Scene Adaptation: It supports a variety of scenes and actions, including complex interactions, dynamic backgrounds, and try-on under different lighting conditions, with strong adaptability.
  • Training on Unpaired Data: It can be trained on unpaired person data, reducing the difficulty and cost of data preparation and improving the model's generalization ability.
  • Complete Outfit Try-On: It supports the try-on of a single piece of clothing and can also achieve coordinated try-on of complete outfits, providing users with a more complete virtual try-on experience.
  • Cross-Domain Applications: DreamVVT can dress cartoon characters in real-world clothing, expanding the application scope of virtual try-on.
  • Dynamic Effect Support: It can generate try-on videos with natural dynamic effects, including clothing fluttering, wrinkle changes, etc., making the try-on effect more realistic.

Technical Principles of DreamVVT

  • Two-Stage Processing Framework: It adopts a two-stage method: the first stage generates high-fidelity keyframe try-on images, and the second stage uses these keyframes to generate a coherent try-on video.
  • Diffusion Transformers (DiTs): It utilizes the Diffusion Transformers architecture, combined with a Vision Language Model (VLM), to achieve high-quality image generation and semantic consistency.
  • Keyframe Sampling and Generation: Representative frames are sampled from the input video, and a multi-frame try-on model is used to generate semantically consistent and high-fidelity keyframe try-on images, providing a foundation for video generation.
  • Skeleton Map and Motion Information Extraction: It extracts skeleton maps and motion information from the input video to guide dynamic changes during video generation and ensure temporal coherence.
  • Adaptation of a Pre-Trained Video Generation Model: It enhances a pre-trained video generation model through a LoRA adapter, combining keyframe try-on images and motion information to generate temporally coherent try-on videos.

DreamVVT Project Address

  • Project Official Website：https://virtu-lab.github.io/
  • Github Repository：https://github.com/Virtu-Lab/DreamVVT
  • arXiv Technical Paper：https://arxiv.org/pdf/2508.02807v1

Application Scenarios of DreamVVT

  • Online Shopping Platforms: Provides virtual try-on functionality for e-commerce platforms. Consumers can upload their own photos or videos to try on clothing of different styles and colors in real time, improving the shopping experience and reducing return and exchange rates.
  • Virtual Fashion Shows: In the fashion industry, it is used to create virtual fashion shows, showcase designers' works, break through the venue and time limitations of traditional fashion shows, and attract more viewers.
  • Entertainment and Film/Television Production: In film and television special effects production, it quickly generates try-on effects of different costumes for characters, saving costume production and filming costs, and allows animated characters to try on real clothing to enhance visual effects.
  • Virtual Character Customization: In the gaming and virtual reality (VR) fields, it provides personalized clothing customization for virtual characters, enhancing users' sense of identity with and participation in virtual characters.
  • Social Media and Content Creation: Users can share fashion outfits on social media through virtual try-on, and creators can use this technology to create interesting content and attract more fans.