What is BlockDance
BlockDance is a new method for accelerating diffusion models jointly launched by Fudan University and the Intelligent Creation team at ByteDance. BlockDance reduces redundant computation and improves inference speed by identifying and reusing structurally similar spatio-temporal features (STSS) across adjacent time steps, achieving up to 50% acceleration. BlockDance focuses on structural features in the later stages of denoising, avoiding image quality degradation caused by reusing low-similarity features. BlockDance introduces BlockDance-Ada, which dynamically allocates computational resources based on reinforcement learning, adjusting the acceleration strategy according to the complexity of different instances, further optimizing content quality and inference speed.
BlockDance's Main Features
- Accelerated Inference Process: By reducing redundant computation, it increases the inference speed of DiTs by 25% to 50%, improving the model's efficiency in practical applications.
- Maintained Generation Quality: While accelerating, it maintains generation results consistent with the original model, ensuring the visual quality, detail representation, and prompt adherence of images and videos.
- Dynamic Resource Allocation: Based on BlockDance-Ada, it dynamically adjusts computational resource allocation according to the complexity of different generation tasks, achieving a better balance between speed and quality.
- Broad Applicability: It supports seamless application to various diffusion models and generation tasks, such as image generation, video generation, etc., with strong generality.
BlockDance's Technical Principles
- Feature Similarity Analysis: During the denoising process of diffusion models, there is high similarity between features at adjacent time steps, especially in the shallow and middle layers of the model. These modules are mainly responsible for generating the structural information of the image, and the structural information is already relatively stable in the early stages of the denoising process. Based on analyzing feature similarity, BlockDance identifies Structurally Similar Spatio-Temporal (STSS) features as the key points for acceleration.
- Caching and Reuse Mechanism: The denoising process is divided into 'cache steps' and 'reuse steps'. In cache steps, the model saves the feature outputs of certain modules in the current step. In subsequent reuse steps, the model directly uses the previously cached features, skipping repeated computation of modules and saving computational resources.
- Dynamic Decision Network (BlockDance-Ada): BlockDance-Ada is introduced, a lightweight decision network based on reinforcement learning. Based on the complexity of the current generation task (such as the structural complexity of the image, the number of objects, etc.), it dynamically determines which steps should be cached and which steps can be reused. The dynamic adjustment mechanism allows BlockDance to achieve a better balance between speed and quality on different generation tasks and models.
- Reinforcement Learning Optimization: The decision network is trained using policy gradient methods in reinforcement learning. By designing a reward function, the trade-off between image quality and computational efficiency is balanced. The reward function includes image quality rewards (such as visual aesthetics, prompt adherence) and computational rewards (such as the proportion of reuse steps). By maximizing the expected reward, the decision network can learn the optimal caching and reuse strategy, achieving maximum acceleration while maintaining generation quality.
BlockDance's Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2503.15927
BlockDance's Application Scenarios
- Image Generation: Accelerates high-quality image generation for artistic creation, game design, etc., while maintaining visual quality.
- Video Generation: Speed up tasks such as video creation and animation production while maintaining visual and temporal consistency.
- Real-time interaction: Used in real-time applications such as VR and AR to respond quickly to user input and improve the experience.
- Large-scale content generation: Efficiently generate large amounts of images and videos, reduce computational costs, and improve efficiency.
- Resource-constrained environments: Run efficiently in resource-limited scenarios such as mobile devices and edge computing without additional training.