ControlNet is the official implementation of the paper "Adding Conditional Control to Text-to-Image Diffusion Models", aimed at adding additional conditional control to diffusion models (such as Stable Diffusion). Its core is a neural network structure that, by copying the weights of neural network blocks into a "locked" copy and a "trainable" copy, lets the trainable copy learn the conditions while the locked copy preserves the original model, so that training on small image-pair datasets will not damage the production-ready diffusion model.
Main capabilities: ControlNet supports a variety of conditional inputs, including Canny edges, M-LSD straight lines, HED boundaries, user scribbles, fake scribbles, human poses, semantic segmentation, depth maps, normal maps, etc. It provides 9 Gradio app demos and supports "guess mode" (non-prompt mode), in which, even if all prompts are removed, the ControlNet encoder can still try to recognize the content of the input control image. In addition, ControlNet can combine multiple models to achieve multi-condition control and supports use on any community model (SD1.X).
Usage: Users need to create a conda environment, download all models and detectors from the Hugging Face page, put the SD models into "ControlNet/models", and put the detectors into "ControlNet/annotator/ckpts". Then they can use it through the provided Gradio app or the A1111 WebUI plugin. Training ControlNet is as easy as training a simple pix2pix, or even easier.
Applicable scenarios: Suitable for scenarios that require precise control over image generation results, such as generating images from edge maps, generating people from poses, and generating scenes from depth maps. It is also suitable for artistic creation, image editing, style transfer, etc.
Limitations: Some models (such as anime line art) have not been released due to potential risks. There are some customization difficulties in the Gradio interface. Low-VRAM mode is suitable for 8GB GPUs or larger batches. Training requires certain computing resources.
