What is IterComp
IterComp is a text-to-image generation framework jointly launched by researchers from Tsinghua University, Peking University, LibAI Lab, the University of Science and Technology of China, the University of Oxford, and Princeton University. Based on an iterative feedback learning mechanism, it aggregates the compositional generation preferences of multiple diffusion models to comprehensively improve the model's overall capability when handling complex compositional tasks. IterComp first builds a gallery containing multiple open-source models, where the models show their respective strengths in different aspects such as attribute binding, spatial relations, and non-spatial relations. Based on the models' preferences, it trains a reward model and uses an iterative optimization strategy to progressively enhance the compositional generation capability of the base diffusion model. This method not only improves the quality and accuracy of generated images but also does not add extra computational overhead, enabling IterComp to surpass existing state-of-the-art methods in multi-category object composition and complex semantic alignment.
Main features of IterComp
- Model preference aggregation: IterComp aggregates the preferences of multiple open-source diffusion models, and the models show their respective strengths in different aspects of compositional generation.
- Dataset construction: Based on model preferences, construct a dataset containing a large number of image ranking pairs for training a composition-aware reward model.
- Iterative feedback learning: Based on an iterative feedback learning method, progressively optimize the base diffusion model and the reward model to improve the model's performance in multi-category object composition and complex semantic alignment.
- Self-optimization: IterComp can self-optimize in a closed loop, continuously improving the quality and accuracy of generated images through multiple iterations.
Technical principles of IterComp
- Model library construction: Select multiple open-source diffusion models that perform excellently in different aspects of compositional generation to form a model library.
- Preference data collection: For key compositionality metrics such as attribute binding, spatial relations, and non-spatial relations, collect preference data from the model library and construct a dataset.
- Reward model training: Use the collected dataset to train reward models targeting different compositionality metrics, and the models will guide the optimization of the base diffusion model.
- Iterative optimization: Based on an iterative feedback learning framework, progressively optimize the base diffusion model and the reward model to achieve continuous self-improvement of the model in compositional generation tasks.
- Effectiveness validation: Based on extensive experiments, validate the effectiveness of IterComp in improving compositional generation capability and compare it with existing state-of-the-art methods.
Project address of IterComp
- GitHub repository：https://github.com/YangLing0818/IterComp
- HuggingFace model library：https://huggingface.co/comin/IterComp
- arXiv technical paper：https://arxiv.org/pdf/2410.07171
Application scenarios of IterComp
- Artistic creation: Artists and designers use IterComp to generate visual artworks with specific styles and complex compositional elements, such as fantasy scenes, character designs, or concept art.
- Game development: In game design, IterComp quickly generates prototype images of game environments, characters, and props, accelerating the creation process of game assets.
- Advertising and brand marketing: Marketers design advertising images, and the images accurately combine multiple elements according to text descriptions to attract the target audience.
- Education and training: In the field of education, IterComp creates illustrations in teaching materials, such as visual explanations of scientific concepts or recreations of historical events.
- Media and Entertainment: Content creators generate concept art for video games, movies, or TV shows, used in the visual elements of storytelling.