What is DanceGRPO
DanceGRPO is the first unified visual generation reinforcement learning framework jointly launched by ByteDance Seed and the University of Hong Kong. It applies reinforcement learning to the field of visual generation, covering two major generation paradigms (diffusion and rectified flow), three tasks (text-to-image, text-to-video, image-to-video), four base models (SD, HunyuanVideo, FLUX, SkyReels-I2V), and five types of reward models (image-video aesthetics, image-text alignment, video dynamic quality, binary reward). DanceGRPO addresses the limitations of RLHF solutions in existing visual generation tasks, achieving seamless adaptation across multiple generation paradigms, tasks, base models, and reward models, significantly improving model performance, reducing VRAM pressure, adapting to training on large prompt datasets, and being transferable to rectified flow and video generation models.
Main Features of DanceGRPO
- Improve visual generation quality: Make generated images and videos more aligned with human aesthetics, more realistic and natural.
- Unify multiple generation paradigms and tasks: Applicable to multiple tasks such as text-to-image, text-to-video, and image-to-video.
- Adapt to multiple models and rewards: Compatible with multiple base models and reward models, meeting diverse needs.
- Improve training efficiency and stability: Reduce VRAM pressure, improve training efficiency, and enhance training stability.
- Enhance human feedback learning capability: Enable models to better learn from human feedback and generate content that better meets human expectations.
Technical Principles of DanceGRPO
- Model the denoising process as a Markov decision process: Model the denoising process of diffusion models and rectified flows as a Markov decision process (MDP), treating the prompt as part of the state and each operation in the denoising process as an action, providing a foundational framework for the application of reinforcement learning.
- Based on SDE sampling equations: To meet GRPO's requirement for stochastic exploration, the sampling process of diffusion models and rectified flows is uniformly formulated in the form of stochastic differential equations (SDE). For diffusion models, the forward SDE describes the process of gradually adding noise to data, and the corresponding reverse SDE is used to generate data. For rectified flows, randomness in the reverse process is achieved by introducing an SDE, providing the necessary stochastic exploration mechanism for reinforcement learning.
- Apply the GRPO objective function for optimization: Drawing on the GRPO strategy from Deepseek-R1, given a prompt, generate a set of output samples and optimize the policy model based on maximizing the GRPO objective function. The objective function considers reward signals and the advantage function among different samples, enabling the model to better learn during training how to adjust the generation strategy according to reward signals, improving the quality of generated results and consistency with human preferences.
- Initialization noise and timestep selection strategy: In the DanceGRPO framework, initialization noise is a key factor. To avoid reward hacking, DanceGRPO assigns shared initialization noise to samples from the same text prompt. DanceGRPO uses a timestep selection strategy to reduce computation while maintaining performance by selecting appropriate optimization timesteps, improving training efficiency.
- Integration of multiple reward models and aggregation of advantage functions: To ensure the stability of training and the high quality of generated results, DanceGRPO uses multiple reward models in practice. Since different reward models may have different scales and distributions, DanceGRPO, based on an advantage function aggregation method, better balances the contributions of different reward models, allowing the model to comprehensively consider evaluation metrics from multiple aspects during optimization and generate visual content that better meets human expectations.
DanceGRPO project address
- Project official website：https://dancegrpo.github.io/
- GitHub repository：https://github.com/XueZeyue/DanceGRPO
- arXiv technical paper：https://arxiv.org/pdf/2505.07818
Application scenarios of DanceGRPO
- Text-to-image generation: Generate high-quality images based on text descriptions, used in fields such as advertising design and game development to improve creative efficiency.
- Text-to-video generation: Generate smooth and coherent videos based on text, suitable for video advertising and educational video production, reducing labor costs.
- Image-to-video generation: Transform static images into dynamic videos, used in animation production and virtual reality to enrich the visual experience.
- Multimodal content creation: Combine text, images, and video to generate diverse content, applied in multimedia education, interactive entertainment, and more, enhancing immersion.
- Creative design and artistic creation: Assist artists and designers in quickly generating creative inspiration and artistic works, stimulating more creativity and improving creative efficiency.