What is OmniBooth
OmniBooth is an image generation framework jointly launched by the Huawei Noah's Ark Lab and a research team from HKUST, supporting spatial control and instance-level customization based on text prompts or image references. The framework uses user-defined masks and associated text or image guidance to precisely control the position and attributes of objects in images, improving the controllability and practicality of text-to-image synthesis technology. The core of OmniBooth lies in an innovative latent control signal, a high-dimensional spatial feature that seamlessly integrates spatial, text, and image conditions to achieve fine-grained image synthesis control.
Main features of OmniBooth
- Multimodal instruction control: Supports controlling image generation with text prompts or image references, achieving image synthesis under multimodal instructions.
- Spatial control and instance-level customization: Users define masks and provide text or image guidance to precisely control the position and attributes of objects in images, achieving instance-level customization.
- High-dimensional latent control signal: Based on the latent control signal, seamlessly integrates spatial, text, and image conditions, providing a unified representation method.
- Flexibility and practicality: Users choose text or images as multimodal conditions as needed, enhancing the flexibility and practicality of generated images.
Technical principles of OmniBooth
- Multimodal embedding extraction：
- Text embedding: Uses a CLIP text encoder to extract embedding vectors of text prompts.
- Image embedding: Uses a DINOv2 feature extractor to extract embedding vectors of image references, preserving the identity and spatial information of images.
- Latent control signal: Draws text and image embedding vectors into a high-dimensional latent control signal; the signal contains spatial information and rich latent features.
- Spatial deformation technology: Uses spatial deformation technology to effectively transform and integrate image embeddings into the latent control signal, preserving the details and structure of images.
- Feature alignment network and edge loss function：
- Develops a feature alignment network to inject conditions into latent features.
- Proposes an edge loss to enhance supervision of high-frequency regions, improving the quality and structural alignment of generated images.
- Multi-scale training and random modality selection strategy: During the training phase, the model uses multi-scale training and a random modality selection strategy to enhance the model's adaptability to inputs of different resolutions and modalities.
Project address of OmniBooth
- Project website：len-li.github.io/omnibooth
- GitHub repository：https://github.com/EnVision-Research/OmniBooth
- HuggingFace model library：https://huggingface.co/lilelife/OmniBooth
- arXiv technical paper：https://arxiv.org/pdf/2410.04932
Application scenarios of OmniBooth
- Dataset generation: Generates synthetic datasets required for training machine learning models, especially when real-world data is difficult to obtain.
- Content creation: Artists and designers create new image content, such as illustrations, concept art, etc., realizing creativity through text or image guidance.
- Games and entertainment: In game development, rapidly generate prototype designs of game environments, characters, and props.
- Virtual reality (VR) and augmented reality (AR): Create realistic backgrounds and objects for virtual environments, enhancing user experience.
- Advertising and marketing: Quickly generate advertising images and marketing materials, customized according to client needs.