What is AnimateDiff-Lightning
AnimateDiff-Lightning is a high-quality video generation model newly launched by researchers at ByteDance, which uses progressive adversarial diffusion distillation technology to achieve fast video generation. The model aims to address the main challenges of existing video generation models in terms of speed and computational cost while maintaining the high quality of generated videos. Compared with the original AnimateDiff, it generates videos more than ten times faster!
AnimateDiff-Lightning official website entry
- Hugging Face model address:https://huggingface.co/ByteDance/AnimateDiff-Lightning
- Hugging Face Demo run:https://huggingface.co/spaces/AP123/AnimateDiffLightning
- arXiv research paper:https://arxiv.org/html/2403.12706v1
AnimateDiff-Lightning features
- Fast video generation: AnimateDiff-Lightning can quickly generate videos through few-step inference, significantly reducing the time required for video generation, making it suitable for scenarios that require instant or fast content generation.
- High-quality output: Despite the fast generation speed, AnimateDiff-Lightning can still maintain the high quality of video content, ensuring that the generated videos have high visual clarity and detail expression.
- Style compatibility: Through cross-model distillation technology, AnimateDiff-Lightning can adapt to a variety of different base models, which means it can generate video content with different artistic styles (such as realism, anime, cartoon, etc.).
- Video generation with different aspect ratios: AnimateDiff-Lightning can generate videos at different aspect ratios and supports a variety of video formats and layout requirements.
- Video-to-video conversion: AnimateDiff-Lightning not only supports text-to-video generation, but can also perform video-to-video style conversion, providing users with a high degree of creative freedom.
- Compatibility with control modules: AnimateDiff-Lightning can be combined with image control modules (such as ControlNet) and camera motion modules (such as Motion LoRA) to further enhance the control capability of video content.
How AnimateDiff-Lightning works
The AnimateDiff-Lightning method is based on improvements to the AnimateDiff model and uses cross-model diffusion distillation technology to improve the speed and quality of video generation. The following are the key steps and components of the AnimateDiff-Lightning method:
1. Model and data preparation
- Select base models: AnimateDiff-Lightning selected multiple popular base models, including realistic-style and anime-style models, such as Stable Diffusion v1.5, RealisticVision v5.1, epiCRealism, ToonYou Beta 6, IMP v1.0, and Counterfeit v3.0.
- Data generation: Use the AnimateDiff model and the selected base models to generate a large amount of video clip data to support the distillation process. This data includes realistic-style videos generated from the WebVid-10M dataset and anime-style videos generated using text prompts.
2. Cross-model distillation
- Shared motion module: The AnimateDiff model consists of a frozen image base model and a shared motion module. During the distillation process, only the weights of the motion module are updated, while the weights of the image base model remain unchanged.
- Multi-GPU training: Load different image base models on different GPUs and initialize the same motion module. This allows distillation on multiple base models simultaneously, improving training efficiency.
3. Flow-Conditioned Video Discriminator
- Extended Discriminator: To handle the different flows of multiple base models, the discriminator is extended to be flow-conditioned, i.e., it can learn and evaluate different flow trajectories based on the index of the base model.
- Training the Discriminator: The discriminator is trained together with the diffusion model and the distillation model to ensure that the generated videos match the original model in style and quality.
4. Distillation Process
- Progressive Distillation: AnimateDiff-Lightning performs distillation in the step order of 128 → 32 → 8 → 4 → 2, using mean squared error (MSE) and classifier-free guidance (CFG) for initial distillation, followed by adversarial loss for subsequent steps.
- Gradient Accumulation: Due to GPU memory constraints, each GPU can only process one batch, so gradient accumulation is used to achieve a larger effective batch size.
5. Evaluation and Testing
- Qualitative Evaluation: By comparison with the original AnimateDiff and AnimateLCM models, the quality of videos generated by AnimateDiff-Lightning under different inference steps is demonstrated.
- Quantitative Evaluation: The FVD (Fréchet Video Distance) metric is used to quantitatively compare videos generated on different base models, to demonstrate the advantage of AnimateDiff-Lightning in maintaining the original quality.