What is ResAdapter
ResAdapter is a resolution adapter designed for diffusion models (such as Stable Diffusion) launched by researchers at ByteDance, allowing these image generation models to generate images with arbitrary resolutions and aspect ratios while maintaining their original style domain. Since diffusion models typically only learn the ability to generate images at specific resolutions during training, when users attempt to generate images beyond the training resolution range, they may not be able to produce high-quality results (such as image collapse, abnormal limbs, etc.). ResAdapter aims to solve this problem. It can expand the resolution range and proportions of images generated by the model without changing the model's original style domain.
ResAdapter Official Website Entry
- Official project homepage:https://res-adapter.github.io/
- GitHub code repository:https://github.com/bytedance/res-adapter
- Hugging Face model:https://huggingface.co/jiaxiangc/res-adapter
- arXiv research paper:https://arxiv.org/abs/2403.02084
ResAdapter Features
- Resolution Interpolation: Allows the model to generate images below its training resolution, enabling the generation of smaller-sized images while maintaining detail and quality.
- Resolution Extrapolation: Enables the model to generate images above its training resolution. This is very important for application scenarios requiring high-resolution output, such as printing, large-size displays, etc.
- Domain Consistency: When generating images at different resolutions, ResAdapter ensures that the image style remains consistent with the style domain during training, avoiding style distortion or inconsistency that may occur when changing resolution.
- Plug-and-Play: ResAdapter is designed to be easily integrated into existing diffusion models without major modifications to the model architecture, and can be quickly applied to various different models and application scenarios.
- Compatibility: ResAdapter is not only compatible with basic diffusion models, but can also be combined with other image generation-related modules (such as ControlNet, IP-Adapter, and LCM-LoRA) to achieve more complex image generation tasks.
How ResAdapter Works
- Analyze Model Structure: First, analyze the UNet architecture of the diffusion model (such as Stable Diffusion) to determine which layers are sensitive to resolution. Typically, convolutional layers are sensitive to resolution because their receptive field is fixed.
- Insert ResCLoRA: Insert Resolution Convolution LoRA (ResCLoRA) into the convolutional layers of the UNet architecture's downsampler and upsampler. ResCLoRA dynamically adjusts the receptive field of the convolutional layers by adding low-rank matrices, enabling them to adapt to input images of different resolutions.
- Introduce ResENorm: To solve the resolution extrapolation problem, introduce Resolution Extrapolation Normalization (ResENorm). ResENorm only trains the group normalization layers in the UNet blocks to adapt to the statistical distribution of high-resolution images while maintaining the model's adaptability to the original style domain.
- Multi-Resolution Training: During the training process, use image datasets of different resolutions for training. This mixed-resolution training strategy allows ResAdapter to learn the ability to generate images at different resolutions while avoiding impact on the original style domain.
- Integrate into Diffusion Models: After training is complete, ResAdapter, as a plug-and-play module, can be integrated into diffusion models of any style. This means that regardless of which style domain the original model focuses on, ResAdapter can extend the resolution range of the images it generates.
- Generated Images: During the inference stage, a diffusion model integrated with ResAdapter can generate images of any resolution according to user needs. The model generates high-quality images through an iterative denoising process based on the input text prompts or conditions (such as the image conditions provided by ControlNet).