What is MagicTryOn
MagicTryOn is a video virtual try-on framework based on the video diffusion Transformer, launched by the College of Computer Science and Technology of Zhejiang University, vivo Mobile Communication, and other institutions.virtual try-onThe framework replaces the traditional U-Net architecture with the more expressive Diffusion Transformer (DiT), and combines a full self-attention mechanism to achieve spatiotemporal consistency modeling of videos. The framework uses a coarse-to-fine garment preservation strategy, based on integrating garment tokens during the embedding stage and introducing multiple conditions such as semantics, texture, and contour lines during the denoising stage, effectively preserving garment details. MagicTryOn demonstrates performance surpassing existing state-of-the-art methods on both image and video try-on datasets, excelling in evaluation metrics, visual quality, and generalization ability in in-the-wild scenarios.
Main features of MagicTryOn
- Garment detail preservation: Accurately simulates the texture, patterns, and contours of garments, maintaining realism and stability when the person moves.
- Spatiotemporal consistency modeling: Ensures coherence between frames in the video, avoids garment flickering and jitter, and provides a smooth try-on experience.
- Multi-condition guidance: Generates more realistic and detailed try-on results based on multiple conditions such as text, image features, garment tokens, and contour line tokens.
Technical principles of MagicTryOn
- Diffusion Transformer (DiT) architecture: The modular design of DiT allows more flexible injection of condition information, enabling multi-level garment features to be effectively integrated into the denoising process. The full self-attention mechanism built into DiT can jointly model the spatiotemporal consistency of videos, capturing intra-frame local details and inter-frame dynamic changes.
- Coarse-to-fine garment preservation strategy：
- Coarse strategy: In the embedding stage, garment tokens are injected into the input token sequence, and the grid size of Rotary Position Embedding (RoPE) is expanded so that garment tokens and input tokens can share consistent positional encoding.
- Fine strategy: In the denoising stage, Semantic-Guided Cross-Attention (SGCA) and Feature-Guided Cross-Attention (FGCA) modules are introduced to provide fine-grained garment detail guidance. SGCA uses text tokens and CLIP image tokens as input to provide a global semantic representation of the garment; FGCA combines garment tokens and contour line tokens to further inject detailed local information.
- Mask-Aware Loss: By introducing Mask-Aware Loss, the model can focus more on the generation of garment regions, improving the detail fidelity of garment regions and the realism of the overall synthesis result.
- Spatiotemporal consistency modeling: Based on the full self-attention mechanism, it jointly models the spatiotemporal consistency of videos, avoiding the limitations of modeling spatial and temporal information separately in traditional methods.
Project address of MagicTryOn
- Project official website：https://vivocameraresearch.github.io/magictryon/
- GitHub repository：https://github.com/vivoCameraResearch/Magic-TryOn/
- arXiv technical paper：https://arxiv.org/pdf/2505.21325
Application scenarios of MagicTryOn
- Online shopping: Users can try on different garments online, improving the shopping experience.
- Fashion design: Designers can quickly preview garment effects, accelerating the design process.
- Virtual fitting room: Provides virtual fitting services for physical stores, reducing the use of physical fitting rooms.
- Advertising and marketing: Brands can create personalized try-on advertisements to attract consumers.
- Gaming and Entertainment: Try on virtual clothing in real time in games to enhance immersion.