stablediffusion-infinity

Purpose

stablediffusion-infinity is used for outpainting with Stable Diffusion on an infinite canvas, that is, extending content beyond the image boundaries. It is powered by the Stable Diffusion inpainting model.

Main capabilities

• Perform outpainting on an infinite canvas; the canvas is infinite by design, but in practice it is limited by RAM and the browser, and may crash or behave abnormally when zoomed out to a certain ratio.
• Provide a Web app based on PyScript and Gradio; the Jupyter Notebook version is on the ipycanvas branch.
• Add photometric correction to suppress seams, requires installing fpie: pip install fpie (Linux/MacOS only).
• Support specifying how to fill blank/masked areas via init_mode; usually patch_match is better than other methods.
• Support disabling the safety_checker, because its false positive rate is relatively high.
• Support adjustment via command-line arguments: python app.py --fp32 --lowvram, for cases where some GPUs are incompatible with fp16.

Usage

• Windows installation guide: setup_guide; Linux installation guide: setup_guide; MacOS installation guide: setup_guide.
• On Windows or Linux with an NVIDIA GPU, you can run with Docker: run_with_docker.
• The canvas requires network access: you can deploy a local HTTP server and serve PyScript, Pyodide, and other JS/CSS resources, and modify index.html accordingly.
• The canvas is implemented using NumPy + PyScript (the project originally used ipycanvas inside Jupyter Notebook), which is less efficient than a pure front-end solution.

Applicable scenarios and limitations

• Applicable scenarios: users who need to extend content beyond image boundaries and create outpainting.
• Limitations: result quality is not guaranteed; prompt engineering, changing the selection size, or reducing the outpainting area may be needed to improve results. Photometric correction may not work (taichi does not support multi-threaded environments; an unreliable subprocess hack has been implemented). When the selection is larger than 512x512, the Stable Diffusion inpainting model becomes much slower. Black squares appear in the results: the safety_checker has a relatively high false positive rate, and safety_checker can be disabled. Some GPUs may be incompatible with fp16.Photometric correction may not work (taichi does not support multi-threaded environments; an unreliable subprocess hack has been implemented). When the selection is larger than 512x512, the Stable Diffusion inpainting model becomes much slower. Black squares appear in the results: the safety_checker has a relatively high false positive rate, and safety_checker can be disabled. Some GPUs may be incompatible with fp16.

Other information

• Pull Requests are welcome to improve UI controls, ideas for better results, or other improvements.
• Regarding why postMessage is not used for iframe interaction: the iframe and gradio are same-origin; for the postMessage version, see the gradio-space version.
• Code source notes: the perlin2d.py code comes from a StackOverflow link and is not covered by this repository's LICENSE; the submodule glid_3_xl_stable is based on Jack000/glid-3-xl-stable; the submodule PyPatchMatch is based on vacancy/PyPatchMatch; the postprocess.py and process.py code is modified based on Trinkle23897/Fast-Poisson-Image-Editing; the convert_checkpoint.py code is modified based on the huggingface/diffusers script; the submodule sd_grpcserver and handleImageAdjustment() in utils.py are based on hafriedlander/stable-diffusion-grpcserver and parlance-zz/g-diffuser-bot; w2ui.min.js and w2ui.min.css come from vitmalina/w2ui, and fabric.min.js is a custom build of fabricjs/fabric.js; interrogate.py is based on pharmapsychotic/clip-interrogator v1, and the submodule blip_model is based on salesforce/BLIP.