Stable Cascade is the official codebase released by Stability AI, providing training and inference scripts as well as a variety of usable models. This model is based on the Würstchen architecture, and its main difference from other models (such as Stable Diffusion) is that it operates in a smaller latent space. It is officially stated that the smaller the latent space, the faster the inference speed and the lower the training cost.
Compression mechanism and efficiency: Stable Diffusion uses 8x compression, encoding a 1024×1024 image into 128×128; Stable Cascade achieves 42x compression, and can encode a 1024×1024 image into 24×24 while maintaining clear reconstruction. The text conditional model is then trained in this highly compressed latent space. Early versions of this architecture achieved a 16x cost reduction compared with Stable Diffusion 1.5. Therefore, this type of model is suitable for uses with high efficiency requirements. It is also officially explained that known extensions such as finetuning, LoRA, ControlNet, IP-Adapter, and LCM can all be used with this method, among which finetuning, ControlNet, and LoRA have already been provided in the training and inference sections.
Model structure: Stable Cascade consists of three models—Stage A, Stage B, and Stage C—forming a cascading process for image generation, which is also the origin of the name "Cascade." Stage A and Stage B are used to compress images, similar to the role of the VAE in Stable Diffusion, but they can achieve higher image compression. Stage C is responsible for generating a small 24×24 latent representation based on the text prompt. Among them, Stage A is a VAE, and Stage B and Stage C are diffusion models.
Model versions: This release provides two Stage C checkpoints, two Stage B checkpoints, and one Stage A checkpoint. Stage C has two versions with 1 billion and 3.6 billion parameters. It is strongly recommended to use the 3.6 billion parameter version, because it has received the most fine-tuning investment. The two versions of Stage B are 700 million and 1.5 billion parameters respectively. Both can achieve good results, but the 1.5 billion parameter version performs better in reconstructing fine details. Therefore, using the larger version of each stage can achieve the best results. Stage A contains 2000M parameters, fixed due to its small size.
Main capabilities and usage: Running the model can be done through the notebook provided in the inference section, which includes more details such as downloading the model, computational requirements, and usage tutorials. Four notebooks are officially provided covering the following use cases:
- A compact notebook providing basic functionality for text-to-image, image variants, and image-to-image. The example prompt is "Cinematic photo of an anthropomorphic penguin sitting in a cafe reading a book and having a coffee.".
- Image variants: The model can understand image embeddings, thereby generating variants of a given image; no prompt is given here. It works in the same way as the regular approach: add noise to the image to a certain degree, then let the model generate from that starting point. In the example, the image on the left is noised to 80%, with the caption "A person riding a rodent.". In addition, the model can also be used in the diffusers library.
- ControlNet: This notebook shows how to use the officially trained ControlNet, or use a ControlNet you trained yourself. This release provides Inpainting / Outpainting and Super Resolution. Face Identity ControlNet will be released later. These can all be used through the same notebook, simply changing the configuration for each ControlNet.
- LoRA: The official implementation is provided for training and using LoRA for Stable Cascade, which can be used to fine-tune the text-conditioned model (Stage C), including adding and learning new tokens, as well as adding LoRA layers to the model. The notebook shows how to use a trained LoRA. For example, after training LoRA on "my dog" with specific training images, given the prompt "Cinematic photo of a dog [fernando] wearing a space suit.", the corresponding image can be generated.
Autoencoder reuse: For those who want to train their own text-conditioned model from scratch (even using an architecture completely different from the official Stage C), they can use the (Diffusion) Autoencoder used by Stable Cascade to work in a highly compressed space. Just as people use Stable Diffusion's VAE to train their own models (such as Dalle3), Stage A and B can likewise be used, while benefiting from higher compression, thereby training and running models faster. The notebook shows how to encode and decode images and the specific benefits. For example, an image batch with dimensions 4 x 3 x 1024 x 1024 can be encoded to a compressed size of 4 x 16 x 24 x 24, with a spatial compression factor of 1024 / 24 = 42.67; afterward, Stage A and B are used to decode back to 4 x 3 x 1024 x 1024. The official statement says the reconstruction results are surprisingly close to small details, and standard VAEs and the like cannot achieve such reconstruction.
Training: The official code for training Stable Cascade from scratch, fine-tuning, ControlNet, and LoRA is provided, with detailed instructions in the training folder.
Installation and running: The official instructions say to first run the following commands to install gradio and diffusers, then run the corresponding commands in the project root directory. The codebase is in an early development stage, and unexpected errors may be encountered or the training and inference code may not be fully optimized. The official team apologizes for this and says that if there is interest, they will continue to release updates introducing the latest improvements and optimizations, and ideas, feedback, or contributions are also welcome.
License: All code in the repository uses the MIT LICENSE; the model weights can be obtained from Hugging Face according to the instructions and are subject to the STABILITY AI NON-COMMERCIAL RESEARCH COMMUNITY LICENSE.
Applicable scenarios and limitations: Suitable for scenarios with high efficiency requirements, such as image generation, text-to-image, image variants, image-to-image, ControlNet (Inpainting / Outpainting, Super Resolution), LoRA fine-tuning, and autoencoder reuse. In terms of limitations, the codebase is in an early development stage and may contain bugs or unoptimized performance; the model weights are under a non-commercial research community license, and the specific commercial terms are not provided in the text; Face Identity ControlNet has not yet been released.
