What is ACE
ACE (All-round Creator and Editor) is an all-round image generation and editing model based on Diffusion Transformer launched by Alibaba Group's Tongyi Lab. ACE introduces the Long-context Condition Unit (LCU) and a unified condition format, can understand and execute natural language instructions, and enables a wide range of visual generation tasks. The ACE model supports multimodal input, can handle complex tasks including image generation, editing, and multi-turn interaction, provides a unified solution, and improves the efficiency and flexibility of visual content creation.
Main Features of ACE
- Multimodal Visual Generation: A generates images according to text instructions, supporting a variety of visual generation tasks, such as style transfer, object addition or removal, etc.
- Image Editing: The model edits existing images, including semantic editing, element editing (such as adding or removing text and objects), and inpainting.
- Long-context Processing: Based on the Long-context Condition Unit (LCU), it understands and executes image editing tasks in multi-turn dialogues, maintaining the coherence of dialogue history.
- Data Collection and Processing: It adopts an efficient data collection method, obtains paired images based on synthetic or clustering pipelines, and uses a fine-tuned large multimodal language model to generate accurate text instructions.
- Single-model Multi-task Processing: It avoids the cumbersome processes used in visual agents and uses a single model backend to respond to any image creation request, improving efficiency.
Technical Principles of ACE
- Long-context Condition Unit (LCU): It introduces LCU, a unified condition format that can combine historical information with current text instructions to better understand user requests and generate the desired images.
- Transformer-based Diffusion Model: It builds a Transformer-based diffusion model. The model uses LCU as input and jointly trains various generation and editing tasks to improve the model's multi-task processing capability.
- Condition Tokenizing: The model encodes text instructions and visual information (such as images and masks) into sequences separately and processes them jointly to achieve alignment of multimodal information.
- Image Indicator Embedding: To ensure that the order of images mentioned in the text instructions matches the image sequence in the CU, predefined text markers are used to indicate the image order.
- Long-context Attention Block: The module distinguishes different spatial and frame-level image embeddings based on timestep embedding (T-Emb) and 3D rotary position encoding (RoPE), ensuring that in the self-attention and cross-attention layers, text embeddings and image embeddings can be aligned frame by frame.
ACE Project Address
- Project Official Website：ali-vilab.github.io/ace-page
- GitHub Repository：https://github.com/ali-vilab/ACE/
- arXiv Technical Paper：https://arxiv.org/pdf/2410.00086
Application Scenarios of ACE
- Art Creation and Design: Artists and designers generate or edit images, realize creative ideas, and improve creative efficiency.
- Media and Entertainment: In film production, generate keyframes or assist in the production of visual effects. In game development, rapidly prototype and generate game assets.
- Advertising and Marketing: Marketers quickly generate attractive advertising images and marketing materials.
- Education and Training: Educators create customized teaching materials and visual aids to enhance students' learning experience.
- E-commerce: E-commerce platforms generate product images, or create personalized product displays based on customer needs.