stable-diffusion

Optimized Stable Diffusion is a modified version of Stable Diffusion designed to reduce GPU VRAM usage at the cost of inference speed. The project is maintained by basujindal, and the code is hosted on GitHub.

Purpose

The main purpose of this project is to allow GPUs with smaller VRAM to run the Stable Diffusion model for tasks such as text-to-image (txt2img), image-to-image (img2img), and inpainting.

Main Capabilities

• Reduced VRAM usage: Through optimizations such as model sharding and chunked attention computation, VRAM requirements are significantly reduced. For example, on an RTX 2060, img2img generating a 512x512 image requires less than 2.4GB of VRAM and takes less than 20 seconds; txt2img under the same VRAM takes less than 24 seconds.
• Support for multiple tasks: Includes txt2img, img2img, and inpainting.
• Provides a command line and Gradio graphical interface: Users can operate through command-line arguments or the Gradio interface.
• Supports seed reproduction: Previously generated images can be reproduced by specifying a seed.
• Supports batch generation: Multiple images can be generated at once, balancing VRAM and speed by adjusting the batch size.
• Supports mixed precision: Mixed precision is enabled by default, and full precision mode can also be used.
• Supports weighted prompts: Different weights can be assigned to words in the prompt.
• Supports multiple samplers: txt2img supports multiple samplers.

Usage

1. Installation: You can clone the entire repository and follow the installation steps of the original Stable Diffusion (mainly creating a conda environment and placing the weight files in the specified location). If you have already cloned the original repository, you only need to download the optimizedSD folder and copy it into the original repository.
2. Docker deployment: Install Docker, the Docker Compose plugin, and the NVIDIA Container Toolkit, clone the repository, place the model files in the specified directory, and then run docker compose up --build.
3. Command-line usage: For example, the command to generate 10 512x512 images: python optimizedSD/optimized_img2img.py --prompt "Austrian alps" --init-img ~/sketch-mountains-input.jpg --strength 0.8 --n_iter 2 --n_samples 5 --H 512 --W 512.
4. Gradio interface: After installing gradio, run python optimizedSD/img2img_gradio.py, python optimizedSD/txt2img_gradio.py, or python optimizedSD/inpaint_gradio.py to launch the corresponding interface.

Applicable Scenarios

• GPU users with limited VRAM, such as GTX series or RTX 2060, etc.
• Scenarios where images need to be generated quickly but VRAM is insufficient.
• Users who need image inpainting functionality.

Limitations

• Slower inference speed: Speed is sacrificed to reduce VRAM.
• Image quality may be affected: Inpainting results are not yet perfect and need to be improved through prompt weighting, prompt engineering, and adjusting the --strength parameter.
• Compatibility issues: GTX series graphics cards may not support mixed precision, resulting in green images, so the --precision full parameter must be used, but this increases VRAM usage.
• Maximum size limit: On a 6GB GPU, images up to about 1152x1088 are supported.
• You need to prepare the model weight files yourself.