What is Wan-Dancer
Wan-Dancer is an open-source music-driven human portrait dance video generation model from Alibaba Tongyi Wanxiang. By inputting a person's photo and a piece of music, it can generate high-quality dance videos with precise rhythm, smooth movements, and distinctive style. The model is the first to break through the minute-level temporal bottleneck, supporting ultra-long coherent output of 720p/30fps from 15 seconds to 3 minutes, covering five dance styles: Chinese classical dance, K-pop, street dance, tap dance, and Latin dance, and supporting personalized dance customization via LoRA.
Wan-Dancer's Main Features
-
Music-Driven Dance Generation: Input a person's photo and music to automatically generate a dance video precisely aligned with the beat.
-
Minute-Level Long Video Generation: Breaks through the traditional 20-second limit, stably generating coherent HD videos from 15 seconds to 3 minutes.
-
Multi-Style Dance Coverage: Supports five significantly different styles: Chinese classical dance, K-Pop, street dance, tap dance, and Latin dance.
-
LoRA Personalized Customization: Only a small amount of specific dance data is needed to train a dedicated motion style and achieve replication of the same dance.
-
Keyframe Secondary Editing: Keyframes generated in the global stage support manual modification, enabling outfit changes and fine-grained motion control.
Wan-Dancer's Technical Principles
-
Global Keyframe Planning (Global DiT): Generates long-duration consistent keyframes based on the complete music structure, plans the dance motion skeleton and key poses, and ensures global temporal coherence.
-
Local Temporal Refinement (Local DiT): Refines motion details and inter-frame transitions based on the global keyframes, solves the problem of monotonous and repetitive movements, and improves the expressiveness of complex motions.
-
Dynamic Frame Rate Adaptation: Introduces RoPE to map absolute time information, achieving precise temporal alignment between music of different durations and dance movements, and eliminating long-sequence drift.
-
Optical Flow Motion Continuity: Optimizes inter-frame transitions based on an optical-flow loss function, maintaining natural coherence in scenes with rapid rotation and complex footwork.
Follow on WeChat and reply "Open Source" to join theAI Open Source Project Exchange Group
How to Use Wan-Dancer
-
Online Experience: Visit the ModelScope Creation Spacehttps://www.modelscope.cn/studios/Wan-AI/Wan-Dancer，上传 9:16 vertical single-person frontal photo and 10–30 seconds of music, select a dance style, and click generate.
-
Local Deployment: Clone the GitHub repository, install the dependency environment, download the 14B weights via ModelScope, and run the global keyframe inference and local temporal refinement scripts separately.
-
Diffsynth-Studio Inference: After installing DiffSynth-Studio, call WanVideoPipeline to load the global/local model configurations, and set parameters such as the reference image, music path, and keyframe mask for generation.
Wan-Dancer's Core Advantages
-
Hierarchical Decoupled Architecture: Global DiT plans long-duration consistent keyframes, while local DiT refines motion details and inter-frame transitions, solving temporal drift and motion repetition problems.
-
Dynamic Frame Rate Adaptation: Introduces RoPE to map absolute time information, ensuring precise synchronization between music of different durations and dance movements.
-
Enhanced Motion Continuity: Optimized with an optical flow-based loss function, maintaining natural coherence in scenarios such as rapid rotation and complex footwork.
-
High-fidelity identity: The identity features of the person in the reference image remain stable over long sequences, avoiding character distortion.
Wan-Dancer's project address
- Project website：https://humanaigc.github.io/wan-dancer-project/
- GitHub repository：https://github.com/Wan-Video/Wan-Dancer
- HuggingFace model library：https://huggingface.co/Wan-AI/Wan-Dancer-14B
- arXiv technical paper：https://arxiv.org/pdf/2607.09581
Comparison of Wan-Dancer with similar competing products
| Comparison dimension | Wan-Dancer | MuseDance |
|---|---|---|
| Research and development institution | Alibaba Tongyi Wanxiang | Stony Brook University + ByteDance + Apple |
| Core architecture | Hierarchically decoupled dual DiT (global keyframe planning + local temporal refinement) | End-to-end U-Net diffusion, based on Stable Diffusion v1.5 |
| Training strategy | The global and local models are trained separately and inferred separately | Two-stage training: the first stage is appearance pretraining, and the second stage freezes spatial attention and injects music/beat/motion modules |
| Music understanding | Dedicated Music Encoder + RoPE temporal mapping | AST (Audio Spectrogram Transformer) extracts music embeddings, which are injected through cross-attention |
| Beat alignment | RoPE dynamic frame rate adaptation, mapping absolute time to the generation process | Librosa extracts beat positions, which are one-hot encoded and then explicitly aligned through cross-attention |
| Motion control | Optical flow loss optimizes inter-frame transitions + Prompt speed annotation | Motion alignment module: uses the hidden states of historically generated frames for temporal self-attention |
| Identity preservation | The reference image is encoded by VAE + global keyframe anchoring constraints | The first stage learns appearance + freezes spatial attention + ReferenceNet feature fusion |
| Generation duration | Minute-level (15 seconds–3 minutes) | Short video clips (experimental setting 4 seconds @ 12fps) |
| Resolution/frame rate | 720p / 30fps | 640×640 / 12fps (experimental configuration) |
Application scenarios of Wan-Dancer
- Short video content creation: Quickly generate personalized dance videos for platforms such as Douyin, Kuaishou, and Xiaohongshu, enabling mass production of diverse dance content without requiring real people to appear on camera.
- Virtual idol and digital human performances: Customize coherent performance materials for specific dance styles for livestreaming, virtual concerts, and digital human accounts, reducing the production cost of dance movements for virtual characters.
- Dance teaching and choreography assistance: With music beat alignment and keyframe preview functions, assist dance teachers in breaking down movement rhythms, designing choreography plans, and generating standardized instructional demonstration videos.
- Advertising and film/TV previsualization: Generate storyboard-level dance shot previews at low cost, helping directors and creative teams quickly validate dance choreography and visual style before shooting, accelerating the decision-making process.