SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI imagesIF

IF

DeepFloyd IF is an open-source text-to-image model launched by StabilityAI's DeepFloyd Lab, with high photorealism and language understanding capabilities. It consists of a frozen T5 text encoder and three cascaded pixel diffusion modules: the base model generates 64x64 pixel images based on text prompts, and two super-resolution modules upscale the images to 256x256 and 1024x1024 pixels, respectively. All stages use a frozen text encoder based on the T5 transformer to extract text embeddings, which are then fed into a UNet architecture enhanced with cross-attention and attention pooling. According to sources, the model achieves a zero-shot FID score of 6.66 on the COCO dataset. The minimum requirements for using all IF models are: IF-I-XL and IF-II-L require 16GB of VRAM, plus Stable x4 super-resolution to 1024x1024 requires 24GB of VRAM, and xformers must be installed and the environment variable FORCE_MEM_EFFICIENT_ATTN=1 must be set. The repository provides local Jupyter Notebook support for Dream, style transfer, super-resolution, and inpainting modes, and integrates with the Hugging Face Diffusers library, allowing stages to be run separately and intermediate results to be inspected. Model cards and weights will be released successively under the DeepFloyd organization on Hugging Face. The initial version uses a restricted license limited to research use only, with plans to release a fully open-source model later. The model has known limitations and biases, as detailed in the model card.

Open website

About this tool

DeepFloyd IF is an open-source text-to-image model launched by StabilityAI's DeepFloyd Lab. According to sources, the model has high photorealism and language understanding capabilities, and is a new type of advanced open-source text-to-image model.

Uses

IF is mainly used for text-to-image generation and supports multiple modes:

  • Dream mode: basic text-to-image generation.
  • Zero-shot image-to-image translation: including style transfer, making the output image exhibit the style of the supporting image.
  • Super-resolution: IF-II and IF-III or Stable x4 can be used to super-resolve images (two-stage cascade), not limited to images generated by IF.
  • Zero-shot inpainting: that is, inpainting. The above modes can be used in local Jupyter Notebooks.

Main Capabilities and Architecture

IF is a modular composite model comprising a frozen text encoder and three cascaded pixel diffusion modules:

  1. Base model: generates 64x64 pixel images based on text prompts.
  2. Two super-resolution models: generate 256x256 pixel and 1024x1024 pixel images, respectively, with resolution increasing progressively. All stages use a frozen text encoder based on the T5 transformer to extract text embeddings, which are then fed into a UNet architecture enhanced with cross-attention and attention pooling. According to sources, the model achieves a zero-shot FID score of 6.66 on the COCO dataset.

Usage

  • Minimum hardware requirements: Using all IF models requires:
    • IF-I-XL (4.3B text-to-64x64 base module) and IF-II-L (1.2B to 256x256 upscaling module) require 16GB of VRAM.
    • Plus Stable x4 (to 1024x1024 upscaling) requires 24GB of VRAM.
    • xformers is required and the environment variable FORCE_MEM_EFFICIENT_ATTN=1 must be set.
  • Local Notebooks: Jupyter Notebooks are provided to support Dream, style transfer, super-resolution, or inpainting modes.
  • Integration with 🤗 Diffusers: IF has been integrated into the Hugging Face Diffusers library. Diffusers allows each stage to be run separately, making it easier for users to customize image generation.process and inspect intermediate results. Before use, you need to accept the terms of use: have a Hugging Face account and be logged in, accept the license on the model card of DeepFloyd/IF-I-XL-v1.0, install huggingface_hub, run the login function in the Python shell and enter the access token.
  • After installing diffusers and its dependencies, you can run the model locally. By default, Diffusers uses model CPU offloading, requiring only about 14 GB of VRAM to run the entire IF pipeline. If using torch>=2.0.0, you need to remove all enable_xformers_memory_efficient_attention() functions.
  • For more methods to accelerate inference and reduce memory consumption, refer to the inference time optimization and low memory optimization sections in the Diffusers documentation.
  • Fine-tuning: Diffusers' dreambooth script supports fine-tuning IF. Through parameter-efficient fine-tuning, new concepts can be added to IF on a single GPU with about 28 GB of VRAM.
  • Running the code locally: load the model into VRAM. Dream mode is text-to-image and supports zero-shot image-to-image translation, super-resolution, zero-shot inpainting, etc.

Use cases

  • Text-to-image generation, pursuing high realism and language understanding.
  • Image style transfer, super-resolution, image inpainting.
  • Custom image generation pipelines and inspection of intermediate results via the Diffusers library.
  • Fine-tuning on a single GPU to add new concepts.

Limitations

  • The model has known limitations and biases; see the model card for details.
  • The initial version uses a restricted license limited to research use, temporarily for collecting feedback, after which a fully open-source model is planned for release.
  • The weights and model card will be released successively under the DeepFloyd organization on Hugging Face, and have their own LICENSE. The code in this repository is released under a custom license.
  • The materials do not provide more information such as specific fees, prices, platforms, effects, or usage steps.

Acknowledgments and team

  • Creators: Alex Shonenkov, Misha Konstantinov, Daria Bakshandaeva, Christoph Schuhmann, Ksenia Ivanova, Nadiia Klokova.
  • Special thanks to StabilityAI and its CEO Emad Mostaque for providing GPU computing and infrastructure, thanks to LAION and Christoph Schuhmann for their contributions to the project and dataset, and thanks to the Huggingface team for optimizing model speed and memory consumption, creating demos, and providing suggestions.
  • External contributors: @Apolinário, @patrickvonplaten, @williamberman, @hysts, @Dango233, etc.
  • The research paper will be released soon.

Related tools

Browse category