What is DanceFusion
DanceFusion is an open-source framework launched by Tsinghua University, focused on audio-driven dance motion reconstruction and generation. DanceFusion combines a hierarchical spatiotemporal Transformer-VAE and a diffusion model, and can handle incomplete and noisy skeleton data from social media to generate realistic dance motions synchronized with music. DanceFusion technology is based on advanced masking strategies and an iterative diffusion process, optimizing motion sequences to ensure high fidelity of motion generation and music synchronization, and is widely used in content creation, virtual reality, interactive entertainment, and other fields.
Main features of DanceFusion
- Audio-driven dance motion reconstruction and generation: DanceFusion generates dance motions synchronized with music based on music, and is suitable for incomplete and noisy skeleton data extracted from social media platforms such as TikTok.
- Handling incomplete and noisy data: The framework can effectively handle missing joints, occlusion, and noise issues, and accurately captures the spatial and temporal information of skeleton sequences based on a hierarchical spatiotemporal VAE.
- Synchronization of audio and motion: Based on the diffusion model, DanceFusion can ensure that dance motions perfectly match the rhythm, melody, and emotion of the music.
- Advanced masking technology: A masking strategy is developed to handle incomplete skeleton data, ensuring that the model only considers reliable joint data during reconstruction.
- Generating high-quality dance motions: The framework can generate high-quality and realistic dance motion sequences with a high degree of diversity and style.
Technical principles of DanceFusion
- Hierarchical spatiotemporal VAE encoding：
- Spatial encoding: Treats each skeletal joint as a token and captures the spatial relationships between joints in the same frame.
- Temporal encoding: Learns the temporal dependencies between frames to ensure the continuity and smoothness of motion sequences over time.
- Diffusion model: Starts from a noisy skeleton sequence and gradually iterates and optimizes it to improve motion realism and synchronization with audio.
- Masking mechanism: Applies a masking mechanism during the encoding stage to mark the presence or absence of joints and prevent the model from considering missing joints.
- Audio feature fusion: Integrates audio features during the iterative process so that the generated motions are precisely synchronized with the music.
- Experimental evaluation: Uses FID and diversity scores to evaluate the quality of generated dance sequences, ensuring the diversity and non-repetitiveness of the output.
DanceFusion project address
- Project official website：th-mlab.github.io/DanceFusion
- arXiv technical paper：https://arxiv.org/pdf/2411.04646
Application scenarios of DanceFusion
- Content creation: Generates dance videos synchronized with music for use in social media and video production.
- Virtual reality (VR) and augmented reality (AR): Provides natural dance motions for virtual characters to enhance immersion.
- Interactive entertainment and games: Generates realistic dance motions in games to improve the gaming experience.
- Dance education and training: Serves as a teaching demonstration to help learners learn and master dance techniques.
- Animation and film production: Generates complex dance motions for virtual characters, reducing the need for motion capture.