What is CreatiLayout
CreatiLayout is an innovative layout-to-image generation(Layout-to-Image, L2I) technology jointly proposed by Fudan University and ByteDance. By building a large-scale layout dataset LayoutSAM, it contains 2.7 million image-text pairs and 10.7 million entity annotations, and each entity details attributes such as color, shape, and texture. The core of CreatiLayout lies in its SiamLayout framework, which treats layout information as an independent modality and, through the MM-DiT native MM-Attention, enables interaction between the layout and image modalities, effectively solving the modality competition problem. In addition, CreatiLayout also introduces LayoutDesigner, which is based on a large language model to help users generate and optimize layouts, and supports multiple input methods, such as center points, masks, sketches, and text descriptions.
Main features of CreatiLayout
- High-quality image generation: Based on the Siamese Multimodal Diffusion Transformer, CreatiLayout can generate high-quality and fine-grained controllable images. It can precisely render complex attributes such as color, texture, and shape.
- Layout generation and optimization: Through LayoutDesigner, CreatiLayout can generate and optimize layouts according to various user inputs, such as center points, masks, sketches, and text descriptions. This enables users to express design intentions more flexibly and generate harmonious and aesthetically pleasing layouts.
- Large-scale dataset support: CreatiLayout builds a large-scale layout dataset named LayoutSAM, containing 2.7 million image-text pairs and 10.7 million entity annotations. Each entity details attributes such as color, shape, and texture, providing rich data support for the model.
- Multimodal interaction: In terms of model architecture, CreatiLayout treats layout information as an independent modality and, through the MM-Attention of MM-DiT, enables interaction between the layout and image modalities.
Technical principles of CreatiLayout
- Siamese Multimodal Diffusion Transformer: CreatiLayout is based on this transformer to achieve high-quality and fine-grained controllable image generation. It can effectively process multimodal data, including images, text, and layout information.
- SiamLayout framework: The framework treats layout information as an independent modality and regards it as equally important as the text and image modalities. Through the MM-DiT native MM-Attention, it enables interaction between the layout modality and the image modality, alleviates the modality competition problem, and enhances the guiding role of layout. Specifically, the interaction among the three modalities of image, text, and layout is decoupled into two Siamese branches: the image-text interaction branch and the image-layout interaction branch, so that text and layout each perform their own roles in guiding image content without interfering with each other.
- LayoutDesigner: Based on a large language model for layout planning and optimization, it supports users in generating and optimizing layouts through multiple input forms, such as center points, masks, sketches, and text descriptions.
Project address of CreatiLayout
- Project official website：https://creatilayout.github.io
- Github repository：https://github.com/HuiZhang0812/CreatiLayout
- arXiv technical paper：https://arxiv.org/pdf/2412.03859
- Online demo：https://huggingface.co/spaces/HuiZhang0812/CreatiLayout
Application scenarios of CreatiLayout
- Promotional poster creation: Designers can use CreatiLayout to quickly generate high-quality promotional posters that meet expectations.
- Furniture placement optimization: CreatiLayout can precisely parse the position and size of each piece of furniture and generate realistic interior renderings.
- Visual effects creation: It can help developers explore new visual effects and styles, and create more creative and attractive game graphics through precise control of layout and image details.
- Teaching material production: Teachers can use CreatiLayout to present knowledge points in an illustrated form.