What Is GCDance
GCDance (Genre-Controlled 3D Full Body Dance Generation Driven by Music) is a 3D dance generation framework introduced by the University of Surrey and Jiangnan University in the UK, capable of generating full-body dance sequences that conform to a specific style based on music and text prompts. GCDance combines high-level music features extracted by pretrained music foundation models (such as Wav2CLIP) and handcrafted music features (such as STFT) to achieve multi-granularity music feature fusion, and uses the CLIP model to embed text prompts into each time step of dance generation, enabling style-controllable dance generation. GCDance supports generating dances of multiple styles under the same music clip and ensures that the dance movements are highly consistent with the rhythm and melody of the music.
GCDance's Main Features
- Style-Controllable Dance Generation: Specify a dance style (such as street dance, jazz dance, etc.) based on text prompts to generate dance movements that conform to a specific style.
- Precise Alignment with Music Rhythm: Dance movements can be highly synchronized with the rhythm, beat, and melody of the music, ensuring a natural match between dance and music.
- Diverse Dance Generation: The same music clip supports generating dances of multiple styles, enriching the expressiveness and diversity of dance.
- Supports Local Editing: Users can customize and modify specific parts of the dance (such as hand movements or specific time segments).
- Generates High-Quality Full-Body Movements: Covers 52 joints (including finger joints), and the generated dance movements are natural, realistic, and highly physically plausible.
GCDance's Technical Principles
- Diffusion Model Framework: Based on a classifier-free diffusion model, it generates dance sequences from noise through step-by-step denoising.
- Music Feature Extraction: It combines pretrained music foundation models (such as Wav2CLIP) to extract high-level semantic features and handcrafted music features (such as Short-Time Fourier Transform, STFT) to capture low-level music details. The multi-granularity feature fusion approach can better capture the complex relationship between music and dance.
- Text Feature Embedding: Based on the CLIP model, text prompts (such as dance style descriptions) are embedded into each time step of dance generation, and a feature adapter is used to align them with the music features, enabling style-controllable dance generation.
- Feature Modulation (FiLM): The Feature-wise Linear Modulation (FiLM) layer dynamically adjusts the dance generation process according to text prompts, ensuring that the generated dance conforms to the specified style.
- Dual-Stream Architecture: Body movements and hand movements are modeled separately and processed independently by two Transformer networks, generating more detailed and expressive full-body dance movements.
- Editing Function: Based on the editing mechanism of the diffusion model (such as diffusion inpainting), users impose constraints on specific parts of the dance during the generation process, enabling local editing and customized generation.
GCDance's Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2502.18309
GCDance's Application Scenarios
- Virtual Reality (VR) and Augmented Reality (AR): Generate virtual character dances to enhance immersion.
- Game Development: Dynamically generate dance movements for game characters to improve interactivity.
- Dance Teaching: Assists with choreography and teaching, providing dance examples in different styles.
- Music Video Production: Automatically generates dance based on music, providing creative material.
- Smart Fitness: Combines music to generate fitness dances, increasing the fun of exercise.