AnimateDiff is an AI animation generation framework, whose official implementation is an ICLR2024 Spotlight paper. The framework is a plug-and-play module that can turn most community text-to-image models into animation generators without additional training.
Purpose AnimateDiff aims to learn transferable motion priors and apply them to other variants of the Stable Diffusion series to achieve text-to-animation generation.
Main capabilities
- Turn community text-to-image models into animation generators without fine-tuning for a specific model.
- Supports Stable Diffusion V1.5 (main branch) and Stable Diffusion XL (sdxl-beta branch).
- Provides sampling scripts and a Gradio demo to facilitate animation generation.
- Supports MotionLoRA, which can adapt to eight basic camera motions (such as zoom, roll, etc.), with each model only 77 MB.
- Supports SparseCtrl, which controls animation content through sparse inputs (such as a small number of RGB images or sketches).
- Technical training is divided into three stages: 1. Mitigate negative effects (train a domain adapter, such as v3_sd15_adapter.ckpt), 2. Learn motion priors (train a motion module, such as v3_sd15_mm.ckpt), 3. Optionally adapt to new modes (train MotionLoRA, such as v2_lora_ZoomIn.ckpt).
Usage
- After setting up the repository and environment, run the sampling script; the generated samples are located in the samples/ folder.
- The default Gradio demo runs at localhost:7860.
- All scripts automatically handle checkpoint downloads, and the first run may take a long time.
- The SDXL beta motion module can be obtained via Google Drive, HuggingFace, or CivitAI. Inference usually requires about 13GB of VRAM and adjustment of hyperparameters (such as sampling steps).
- Community resources: Stable Diffusion WebUI plugin sd-webui-animatediff, ComfyUI plugin ComfyUI-AnimateDiff-Evolved, Google Colab.
- Officially supports Diffusers; refer to the AnimateDiff Diffusers Tutorial.
Applicable scenarios
- Text-to-animation generation.
- Imageanimation/interpolation (recommended to use images generated by the same community model).
- Style alignment and personalized animation.
Limitations
- Academic use only.
- Slight flickering exists.
- For compatibility with community models, no specific optimization was made for general text-to-video, resulting in limited visual quality under this setting.
- The official website only https://github.com/guoyww/AnimateDiff and https://animatediff.github.io，其他网站均不相关。
- Responsibility for user-generated content lies with the user.
