What is Diffusion Self-Distillation
Diffusion Self-Distillation (DSD) is a zero-shot customized image generation technology launched by Jiajun Wu's team at Stanford University. It uses a pretrained text-to-image diffusion model to automatically generate a dataset and fine-tunes it into a model capable of text-conditioned image-to-image tasks. Diffusion Self-Distillation is based on generated image grids and vision-language model filtering to create high-quality paired datasets, thereby enabling identity-preserving customized image generation for any instance in any context without human intervention. This method performs excellently on identity consistency preservation tasks, competing with per-instance tuning techniques, and requires no test-time optimization.
Main features of Diffusion Self-Distillation
- Identity-preserving customized image generation: Generate images of a specific instance in different contexts while preserving the identity features of that instance.
- Zero-shot learning: Directly apply a pretrained model to generation tasks without training data for a specific instance.
- Automated data pairing: Automatically create high-quality training data pairs with the assistance of a self-generated dataset and a vision-language model.
- Image-to-image translation: Support converting an input image into an output image with specific text conditions, such as changing lighting, style, or other visual attributes.
- Broad applicability: Suitable for a variety of text-conditioned image generation tasks, including personalization, relighting, depth control, and instruction following.
Technical principles of Diffusion Self-Distillation
- In-context generation capability of pretrained models: Create image grids based on the in-context generation capability of pretrained text-to-image diffusion models.
- Data pairing and filtering: Use vision-language models (VLMs) to assist filtering and create large-scale, high-quality paired datasets from the generated image grids.
- Fine-tuning the diffusion model: Fine-tune the pretrained text-to-image diffusion model with the filtered paired dataset, transforming it into an image generation model that supports text + image conditions.
- Parallel processing architecture: Introduce a parallel processing architecture that treats the input image as the first frame of a video sequence and generates a two-frame video as output. The first frame reconstructs the input image, and the second frame is the edited output, enabling effective information exchange.
- Information exchange and editing: Establish identity mapping and conditional editing objectives between the two video frames, enabling the model to capture complex semantics and perform complex edits.
Project address of Diffusion Self-Distillation
- Project website：primecai.github.io/dsd
- arXiv technical paper：https://arxiv.org/pdf/2411.18616
Application scenarios of Diffusion Self-Distillation
- Artistic creation: Artists create works in different styles and backgrounds while maintaining the consistency of characters or objects in the artwork.
- Game development: In game design, quickly generate multiple variants of game characters or items with consistent identity features to adapt to different game scenes.
- Film and animation production: Filmmakers maintain character consistency across different scenes, or re-render scenes under different lighting conditions.
- Advertising and marketing: Marketers customize advertising images to ensure brand image consistency across various advertising media.
- Personalized products: Generate personalized products based on user-uploaded images, such as custom T-shirts, mugs, and phone cases, while maintaining consistency of brand elements.