What Is MultiBooth
MultiBooth is a multi-concept image generation method launched by Tsinghua Shenzhen International Graduate School, Meta, The Hong Kong University of Science and Technology, and other institutions. It can generate images containing multiple user-specified concepts from text. MultiBooth divides the generation process into two stages: single-concept learning and multi-concept integration. In the single-concept learning stage, based on a multimodal image encoder and Adaptive Concept Normalization, it learns a concise and discriminative embedding representation for each concept, and uses LoRA technology to improve concept fidelity. In the multi-concept integration stage, it uses a Region Customization Module (RCM) to generate each concept within specified regions according to bounding boxes and regional prompts, and ensures accurate interaction between different concepts based on the base prompt. While maintaining high image fidelity and text alignment capability, MultiBooth achieves efficient multi-concept image generation and has low costs in both training and inference stages.
Main Features of MultiBooth
- Multi-Concept Image Generation: Generates images containing multiple specified concepts based on user-provided text prompts.
- High Fidelity and Text Alignment: The generated images have high fidelity, clearly showing the detailed features of each concept, and are highly aligned with the text prompts, ensuring that the image content is consistent with user intent.
- Efficient Inference: In the multi-concept generation process, inference cost is low, and inference time does not increase significantly as the number of concepts increases, making multi-concept image generation more efficient.
- Plug-In Generation: Supports combining different single-concept modules in a plug-in manner for multi-concept image generation, without retraining the model for each concept combination, improving the model's flexibility and scalability.
Technical Principles of MultiBooth
- Single-Concept Learning Stage：
- Multimodal Image Encoder: Uses the QFormer encoder, taking images and concept names (such as "dog") as input, and based on the interaction of self-attention layers and cross-attention layers, generates customized embedding representations aligned with text, learning a concise and discriminative embedding for each concept.
- Adaptive Concept Normalization (ACN): Adjusts the L2 norm of the customized embeddings to make them comparable to other word embeddings in the prompt, solving the inter-domain discrepancy problem in the embedding space and improving multi-concept generation capability.
- Efficient Concept Encoding Technology: Uses LoRA technology to perform low-rank decomposition on the attention layers in U-Net, avoiding language drift caused by fine-tuning U-Net, improving concept fidelity in single-concept learning, and reducing additional parameter storage requirements.
- Multi-Concept Integration Stage：
- Region Customization Module (RCM): In the cross-attention layers, according to bounding boxes and regional prompts obtained by user definition or automated processes, divides image features into different regions. Each region is guided by the corresponding single-concept module and prompt to generate concepts, and based on the base prompt ensures interaction between concepts in different regions, achieving accurate fusion of multiple concepts in the same image.
- Parallel Generation and Interaction: In RCM, multiple single-concept modules can generate simultaneously, and based on the cross-attention mechanism, achieve parallel interaction between concepts, avoiding concept fusion and increases in inference cost.
MultiBooth Project Address
- Project Website：multibooth.github.io
- GitHub Repository：https://github.com/chenyangzhu1/MultiBooth
- arXiv technical paper：https://arxiv.org/pdf/2404.14239
Application scenarios of MultiBooth
- Entertainment and creative industries: Quickly generate mysterious ancient tomb scenes for adventure games, showcasing internal mechanisms and murals, enriching game exploration elements.
- Advertising and marketing: Create cosmetic mask advertising posters showing young women's facial skin looking refreshed after use, conveying product efficacy and brand positioning.
- Education and learning: Generate images of medieval castles, clearly displaying tower and city wall structures, helping students understand castle characteristics and deepen their memory of historical knowledge.
- E-commerce: Generate outfit images for summer dresses, showing the effect of floral dresses paired with woven sandals and straw hats, attracting customers to purchase.
- Scientific research and engineering: Generate schematic diagrams of new nanomaterial structures, showcasing their ultra-light weight and high-strength properties, helping the public understand the innovativeness of scientific research achievements.