"The Illustrated Stable Diffusion" is a technical analysis article written by Jay Alammar, aiming to introduce the working principles of Stable Diffusion in a visual way. The article points out that Stable Diffusion is a system composed of multiple components and models, not a single model.
The article first focuses on the text-to-image (text2img) mode, and explains that Stable Diffusion can also be used for image modification (input is text plus image). Then, the article breaks down in detail the three core components:
-
Text encoder (ClipText): a Transformer-based language model that converts input text into numerical representations. Specifically, it outputs 77 token embedding vectors, each with a dimension of 768.
-
Image information generator (UNet + scheduler): processes information step by step in latent space, which is the key to Stable Diffusion's performance improvement. This component runs for multiple steps, corresponding to the steps parameter in the interface, usually defaulting to 50 or 100.
-
Image decoder (Autoencoder Decoder): takes the processed information array from the information generator and runs only once to generate the final pixel image. The input dimension is (4,64,64), and the output dimension is (3, 512, 512) (that is, red/green/blue, width, height).
The article also explains the Diffusion process: training samples are created by gradually adding noise to images, a noise predictor is trained, and then images are generated by gradually removing noise from the noise through the reverse process. This process takes place in latent space rather than pixel space, thereby accelerating generation.
In addition, the article discusses how to incorporate text information into the generation process: by combining the output of the text encoder with the attention layers in the UNet, the model can control image generation according to text prompts.
The article provides version update notes (the V2 version in November 2022 updated the image description of forward diffusion and added images), and mentions that there are Chinese and Vietnamese translations.
