DiffusionGPT - An open-source large-model-driven text-to-image generation system

What is DiffusionGPT?

DiffusionGPT is an open-source large language model (LLM)-driven text-to-image generation system launched by researchers from ByteDance and Sun Yat-sen University, aiming to address the challenge in the text-to-image field of being unable to handle different inputs or being limited to the results of a single model. The system uses Tree-of-Thought and Advantage Database technologies to handle multiple types of text prompts and combine these prompts with domain expert models to generate high-quality images.

DiffusionGPT official website entry

• Official project homepage:https://diffusiongpt.github.io/
• Arxiv research paper:https://arxiv.org/abs/2401.10061
• GitHub code repository:https://github.com/DiffusionGPT/DiffusionGPT
• Hugging Face running address:https://huggingface.co/spaces/DiffusionGPT/DiffusionGPT
• DiffusionGPT-XL Demo：https://huggingface.co/spaces/DiffusionGPT/DiffusionGPT-XL

Main features of DiffusionGPT

• Text prompt parsing: DiffusionGPT can understand and parse various types of text prompts, including description-based, instruction-based, inspiration-based, and hypothesis-based prompts. This capability enables the system to accurately grasp the image content the user wants to generate.
• Model selection and integration: The system classifies and organizes multiple domain expert generation models by building a structure based on Tree-of-Thought (ToT). This allows DiffusionGPT to select the most suitable one from numerous models to generate images according to the input text prompt.
• Human feedback optimization: DiffusionGPT uses human feedback to optimize the model selection process. Through Advantage Databases, the system can select the best-performing model based on human ratings of model-generated results, thereby improving the quality of generated images and user satisfaction.
• Image generation execution: After selecting a suitable model, DiffusionGPT executes the image generation process. To enhance the detail and artistry of the generated images, the system also enriches and refines the input prompt through a prompt expansion agent.
• Multi-domain applicability: DiffusionGPT is designed as an all-purpose system that is not only suitable for descriptive text prompts but can also handle more complex instructions and inspirational content, giving it broad applicability in diverse application scenarios.
• Plug-and-play solution: DiffusionGPT is designed to be a training-free, easy-to-integrate solution that can be easily integrated into existing image generation workflows, providing users with convenient services.

How DiffusionGPT works

The working principle of DiffusionGPT can be divided into four main steps, which work together to achieve the process from text prompt to high-quality image generation:

1. Prompt Parse：
   • DiffusionGPT first uses a large language model (LLM) to analyze and extract key information from the input text prompt. This process is crucial for generating the content expected by the user, because user input may contain multiple types of prompts, such as description-based, instruction-based, inspiration-based, or hypothesis-based.
   • The LLM can identify the different forms of these prompts and extract the core content, so as to provide accurate guidance for subsequent image generation.
2. Tree-of-Thought of Models：
   • After parsing the prompt, the system builds a structure based on a Tree of Thought (ToT), which contains multiple domain expert generation models. These models are classified into different nodes according to their attributes, forming a hierarchical structure.
   • Through this Tree of Thought, the system can narrow down the range of candidate models and improve the accuracy of model selection. This process is similar to searching in a tree, starting from the root node and gradually moving downward based on the prompt content to find the best-matching model.
3. Model Selection：
   • After determining the set of candidate models, DiffusionGPT uses human feedback and Advantage Databases to select the most suitable model. This database contains scores for the generation results of the models, and based on these scores, the system can determine which models perform best when handling specific types of prompts.
   • The system calculates semantic similarity between the input prompt and the prompts in the database, and then selects the model that best matches these prompts to ensure that the generated image meets the user's expectations.
4. Execution of Generation：
   • Finally, the selected model generates an image based on the extracted core prompt. To improve the quality of the generated image, DiffusionGPT also uses a Prompt Extension Agent to enrich and refine the input prompt, making it more detailed and descriptive.
   • In this way, the generated image can not only capture the core content of the prompt, but also display higher detail and artistry.

Through these four steps, DiffusionGPT can seamlessly process diverse text prompts and generate high-quality images that are highly consistent with user intent. The design of this system aims to improve the flexibility and efficiency of image generation while using human feedback to continuously optimize the generation process.