What is EliGen
EliGen is a novel entity-level controllable framework jointly developed by Zhejiang University and Alibaba Group.Image GenerationBy introducing a regional attention mechanism, it seamlessly integrates entity prompts and arbitrarily shaped spatial masks into a diffusion transformer without additional parameters. EliGen includes a dataset of 500,000 high-quality annotated samples for training the model to achieve robust and accurate entity-level manipulation. EliGen proposes an inpainting fusion pipeline that can be extended to multi-entity image inpainting tasks.
Main Features of EliGen
- Precise Entity-Level Control: Through the regional attention mechanism, EliGen can precisely control each entity in the image, including the entity's position, shape, and semantic attributes.
- Multi-Entity Image Inpainting: EliGen proposes an inpainting fusion pipeline that can achieve multi-entity image inpainting tasks. Through the regional attention mechanism, multiple entities can be inpainted in a single forward pass.
- Stylized Entity Control: Combined with IP-Adapter, EliGen can generate target images according to the style of a reference image, achieving stylized entity control.
- Interactive Image Design and Editing: Through integration with MLLM (Multimodal Large Language Model), EliGen can perform image design and editing based on dialogue, and users can generate or modify images through text descriptions.
- Integration with Community Models: EliGen can seamlessly integrate with open-source models such as IP-Adapter, In-Context LoRA, and MLLM, unlocking new creative possibilities. It can achieve stylized entity control, entity transfer, and dialogue-based image design and editing.
- Strong Generalization Ability: EliGen maintains good generalization ability under different random seeds, continuously changing entity positions, and unreasonable position inputs, generating high-quality images that meet requirements.
Technical Principles of EliGen
- Regional Attention Mechanism: EliGen introduces a regional attention mechanism, which extends the attention module of the Diffusion Transformer (DiT) and can handle entity masks of arbitrary shapes. EliGen combines global prompts and local prompt embeddings to form an extended prompt sequence, and constructs a joint attention mask (including entity-latent, inter-entity, and intra-entity masks) based on spatial conditions. Without additional parameters, it can modify entity details before training.
- High-Quality Dataset Construction: To train EliGen, the researchers constructed a dataset containing 500,000 high-quality annotated samples. The samples are generated by Flux, and global prompts and entity information are annotated based on the Qwen2-VL vision-language model. The high-quality dataset ensures the robustness and accuracy of the model in entity-level manipulation.
- Training and Fine-Tuning: EliGen adopts the LoRA (Low-Rank Adaptation) method for efficient fine-tuning, ensuring fast convergence of the model. LoRA weights are applied to the linear layers of each block of the DiT, including the projection layers of the attention layers and the linear layers within the adaptive LayerNorm.
- Inpainting Fusion Pipeline: EliGen proposes an inpainting fusion pipeline for multi-entity image inpainting tasks. The pipeline uses region-based noise fusion operations, enabling EliGen to inpaint multiple entities in a single forward pass while maintaining the quality of non-inpainted regions.
EliGen Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2501.01097
EliGen Application Scenarios
- Virtual Scene Generation: EliGen can generate high-quality virtual scenes and supports entity control with masks of arbitrary shapes, making it suitable for scene design in virtual reality and game development.
- Character and Prop Design: Users can precisely control the position, shape, and style of characters and props, providing a powerful tool for character and prop design in virtual worlds.
- Large-Scale Customized Data Synthesis: EliGen's arbitrary-shape mask control capability enables it to efficiently generate large-scale customized data, making it suitable for data augmentation and machine learning tasks.
- Product Display and Promotion: EliGen can generate high-quality product display images and supports precise entity control and stylized design, meeting the needs of advertising and product design.