What is InstanceAssemble
InstanceAssemble is a lightweight layout-to-image generationframework jointly launched by Xiaohongshu and Fudan University. Through an innovative "instance assembly attention" mechanism, the framework achieves precise image generation from simple to complex and from sparse to dense layouts. Users only need to provide the bounding box positions and content descriptions of objects, and the AI can generate images that match the semantics at the corresponding positions. InstanceAssemble is based on the diffusion transformer architecture and adopts a lightweight adaptation approach. It requires only a small number of additional parameters to adapt to mainstream models, greatly lowering the barrier to use.
Main Features of InstanceAssemble
-
Precise Layout Control: By specifying the position (bounding box) and content description of each object, the AI generates image content that matches the semantics at the corresponding positions.
-
Layout Generation from Simple to Complex: For simple scenes with a few objects as well as complex, dense scenes, InstanceAssemble can maintain high-precision layout alignment and semantic consistency.
-
Multimodal Content Control: It supports defining the content of each instance through multiple modalities such as text descriptions, reference images, depth maps, and edge maps, further improving the accuracy and detail expression of the generated images.
-
Lightweight Adaptation: There is no need to retrain the entire model. With only a small number of additional parameters (for example, adapting the Stable Diffusion 3-Medium model requires only about 3.46% additional parameters), it adapts to a variety of mainstream diffusion models, lowering the barrier to use.
-
Strong Generalization Ability: During training, only sparse layouts (≤10 instances) are used, yet it can maintain robust performance on dense layouts (≥10 instances), adapting to layout conditions of different complexity.
Technical Principles of InstanceAssemble
-
Diffusion Model Foundation: InstanceAssemble is based on the current mainstream diffusion transformer architecture (such as Multimodal Diffusion Transformer, MMDiT), using the powerful generation capability of diffusion models to generate high-quality images through a gradual denoising process.
-
Instance Assembly Attention Mechanism：
-
Layout Encoder: It encodes the layout conditions provided by the user (such as bounding boxes and text descriptions) into instance tokens. For each instance, it combines its position information (enhanced through DenseSample) and content description (text or visual content) to generate the corresponding instance token.
-
Assemble-Attn Module: During the generation process, image tokens interact with instance tokens. For each instance, it attends only to the image region within the bounding box and updates the region features through the attention mechanism. The updated features are fused into the overall image through weighted assembly, ensuring the layout and semantic consistency of each instance.
-
Cascaded Structure: It adopts a cascading mechanism, using the base MMDiT model to process global text prompts and image features, and then using the Assemble-MMDiT module to process instance layout conditions, achieving precise layout control while maintaining global generation quality.
-
-
Lightweight Adaptation (LoRA): To efficiently adapt to existing diffusion models, InstanceAssemble uses Low-Rank Adaptation (LoRA) technology, introducing only a small number of trainable parameters (such as low-rank matrices) in the attention modules, enabling layout control functionality without changing the capabilities of the base model.
-
Evaluation and Benchmarking: To more accurately measure the degree of matching between layouts and images, InstanceAssemble proposes the "Layout Grounding Score" (LGS) evaluation metric and creates the "DenseLayout" benchmark dataset containing 5,000 images and 90,000 instances, for evaluating generation performance under complex layout conditions.
InstanceAssemble Project Address
- GitHub Repository：https://github.com/FireRedTeam/InstanceAssemble
- arXiv Technical Paper：https://arxiv.org/pdf/2509.16691
InstanceAssemble Application Scenarios
-
Design and Advertising: Helps designers and advertising agencies quickly generate design sketches or advertising visuals that conform to specific layouts and styles, with precise control over element positions and content.
-
Content Creation: Provides content creators with a high-quality image generation tool for social media, video production, etc., enhancing content appeal and professionalism.
-
Game Development: The framework can quickly generate game scenes and character equipment layouts, helping game developers efficiently design realistic backgrounds and levels.
-
Education and Training: The framework can generate teaching materials and virtual laboratory scenes, helping teachers convey knowledge more intuitively and improving teaching effectiveness.
-
Architectural Design: Assists interior designers and architects in generating layout drawings and renderings of interior spaces and building exteriors, optimizing design plans.