What is BrushNet
BrushNet is a plug-and-play image photo inpainting model based on diffusion models, launched by the ARC Lab of Tencent PCG and researchers from the University of Hong Kong. It effectively handles masked regions in images through a decomposed dual-branch architecture. One branch of the model focuses on extracting pixel-level features of the masked image, while the other branch is responsible for image generation. This design enables BrushNet to finely integrate key mask information into the inpainting process in a hierarchical manner, thereby generating high-quality inpainting results while maintaining the coherence of the original image content.
Compared with previous image inpainting methods (such as Blended Latent Diffusion, Stable Diffusion Inpainting, HD-Painter, PowerPaint, etc.), BrushNet's image restoration and inpainting capabilities demonstrate superior coherence in aspects such as style, content, color, and prompt alignment.
BrushNet official website entry
- Official project homepage:https://tencentarc.github.io/BrushNet/
- GitHub code repository:https://github.com/TencentARC/BrushNet
- arXiv research paper:https://arxiv.org/abs/2403.06976
BrushNet features
- Restore different types of images: BrushNet can restore images from different scenes, such as humans, animals, indoor and outdoor scenes, and can also restore images in different styles, such as natural images, pencil drawings, anime, illustrations, watercolors, etc.
- Pixel-level inpainting: BrushNet can identify and process masked regions in images, performing precise inpainting on each pixel to ensure that the restored region seamlessly connects with the original image visually.
- Preserve unmasked regions: Through hierarchical control and a specific blur fusion strategy, BrushNet can preserve unmasked regions during the inpainting process, avoiding unnecessary changes to the original image content.
- Compatibility with pre-trained models: As a plug-and-play model, BrushNet can be combined with various pre-trained diffusion models (such as DreamShaper, epiCRealism, MeinaMix, etc.), leveraging the powerful generative capabilities of these models to complete inpainting tasks.
- Flexibility and controllability: Users can control the scale and details of inpainting by adjusting the model's parameters, including the size of the inpainted region and the level of detail of the inpainted content.
How BrushNet works
BrushNet performs image inpainting tasks based on diffusion models through an innovative dual-branch architecture.
The following is a brief overview of how BrushNet works:
- Dual-branch architecture: The core of BrushNet is a decomposed dual-branch architecture, in which one branch focuses on processing the features of the masked image, while the other branch is responsible for generating the rest of the image.
- Masked image feature extraction: In the masking branch, the model uses a variational autoencoder (VAE) to encode the masked image in order to extract its latent features. These features are then used to guide the image inpainting process.
- Pre-trained diffusion model: In the generation branch, the model uses a pre-trained diffusion model to generate image content. This model has already learned how to recover a clear image from noise.
- Feature fusion: The extracted masked image features are gradually fused into the pre-trained diffusion model, so that the inpainting process can be finely controlled in a hierarchical manner.
- Denoising and generation: In the reverse diffusion process, the model progressively recovers a clear image from noise through iterative denoising steps. Each step takes into account the features of the masked image to ensure that the inpainted region remains visually consistent with the rest of the original image.
- Blur Fusion Strategy: To better preserve the details of the unmasked regions, BrushNet adopts a blur fusion strategy. This means that when fusing the masked region with the generated region, a blurred mask is used to reduce hard edges and unnatural transitions.
- Output Inpainted Image: Finally, the model outputs an inpainted image in which the masked region is filled naturally and coherently, while the original content of the unmasked regions is preserved.