LDGen - Li Auto's Multilingual Text-to-Image Generation Technology

What is LDGen

LDGen is an innovativetext-to-imagesynthesis technology that improves the quality and semantic consistency of text-description-to-image generation by combining large language models (LLMs) with diffusion models. Through hierarchical caption optimization and human instruction techniques, it extracts precise semantic information from text, and based on a lightweight adapter, it achieves efficient alignment and interaction between LLMs and image features. LDGen supports zero-shot multilingual generation and can generate high-quality images from text descriptions in multiple languages, significantly outperforming traditional methods.

LDGen's Main Features

• Multilingual Zero-Shot Generation: By combining large language models (LLMs) with existing text-to-image diffusion models, LDGen supports zero-shot multilingual text-to-image generation. Even when trained only on English prompts, the model can generate images described in multiple languages, significantly improving cross-lingual generation capability.
• Language Representation Optimization: It adopts hierarchical caption optimization and human instruction techniques to extract more precise semantic information and enhance semantic alignment between text and images. It improves the semantic consistency of generated images and avoids false information caused by incorrect instructions.
• Improved Generation Quality: In multiple experiments, LDGen outperforms baseline models and other enhancement methods such as PixArt-, ELLA, etc., in terms of instruction following and image aesthetic quality.

LDGen's Technical Principles

• Language Representation Strategy: LDGen adopts hierarchical caption optimization and human instruction techniques to extract more precise semantic information. By generating captions of different lengths and combining them with human instruction optimization, the model can better capture the hierarchical structure of image content while avoiding false information caused by incorrect instructions.
• LLM Alignment Module: To align the features of the LLM with those of existing diffusion models, LDGen designs a lightweight adapter. The adapter adjusts the feature space output by the LLM to match the feature space of text encoders such as T5, achieving efficient feature alignment.
• Cross-Modal Refiner: LDGen introduces a cross-modal refiner module to enhance the interaction between LLM features and image features. Through components such as self-attention mechanisms and cross-attention mechanisms, the module optimizes the LLM's feature representation and further improves semantic alignment between text and images.
  • Self-Attention Mechanism (Self-Attention): Optimizes the internal representation of LLM features.
  • Cross-Attention Mechanism (Cross-Attention): Uses LLM features as the query (Query) and image features as the key (Key) and value (Value), promoting deep interaction between text and images.
  • Learnable Scaling Factors (Learnable Scaling Factors): Dynamically balances the original features and the optimized features during training, ensuring a seamless transition from pretrained weights to new features.
• Efficient Training Strategy and Efficiency: LDGen significantly reduces computational requirements through staged training:
  • Feature Alignment Stage: Uses about 80 million text data entries to train the LLM alignment module.
  • Fine-Tuning Stage: Fine-tunes at 512 resolution using 24 million text-image pairs.
  • High-Resolution Training Stage: Continue training with 14 million data entries at 1024 resolution. The entire training process requires only about 120 A100 GPU days, reducing computational resources by about 74% compared to PixArt-α.

LDGen's project address

• Project website：https://zrealli.github.io/LDGen/
• Github repository：https://github.com/zrealli/LDGen
• arXiv technical paper：https://arxiv.org/pdf/2502.18302

LDGen's application scenarios

• Artistic creation and design: Artists and designers can use LDGen to generate high-quality images from creative descriptions, accelerating the creative process. Quickly transforming text descriptions into visual content helps creators explore different design directions.
• Advertising and marketing: In the fields of advertising and marketing, LDGen can quickly generate attractive advertising images and social media posts based on brand style or market trends. Directly generating images from text descriptions can improve the appeal and personalization of promotional materials.
• Media and entertainment: LDGen can be used for concept art creation in film, game, and animation production, generating preliminary visual representations of scenes and characters. In film production, preliminary sketches of special effects scenes can be generated, helping directors and designers quickly preview scene layouts.
• Education: In the field of education, LDGen can help students and teachers create teaching materials, such as recreations of historical scenes or visualizations of scientific concepts. Generating images from text descriptions can display complex concepts more intuitively.
• E-commerce: Online retailers can use LDGen to generate visual displays of products, showing how clothing looks in different environments or on different models. It can help merchants quickly generate high-quality product images and improve the user experience.