What is LongAlign
LongAlign is an improved method for text-to-image (T2I) diffusion models launched by a research team at the University of Hong Kong, capable of improving the alignment accuracy of long text inputs. LongAlign uses segment-level encoding technology to split and process long text, adapting to the input limitations of encoding models. At the same time, it introduces decomposed preference optimization, which applies different weights based on distinguishing text-related and text-unrelated parts in the preference model to reduce overfitting and enhance alignment. After 20 hours of fine-tuning, LongAlign significantly improves the performance of the Stable Diffusion v1.5 model on long text alignment tasks, surpassing advanced models such as PixArt-α and Kandinsky v2.2.
LongAlign's main features
- Long text processing: Based on a segment-level encoding method, it processes long text inputs and overcomes the maximum input length limit of pre-trained encoding models such as CLIP.
- Text-to-image alignment: Improves the alignment between generated images and input text, ensuring the accuracy of image content relative to the text description.
- Reducing overfitting: Based on preference decomposition and a reweighting strategy, LongAlign reduces overfitting during the fine-tuning process and improves the model's generalization ability.
LongAlign's technical principles
- Segment-level encoding: Splits long text into multiple segments (or sentences), encodes each segment independently, and merges the encoding results. This supports the model in processing text that exceeds the maximum input length limit.
- Preference decomposition: Analyzes the scoring mechanism of the preference model and decomposes the preference score into two parts: the text-related part (measuring text-to-image alignment) and the text-unrelated part (evaluating other visual aspects of the image, such as aesthetics).
- Reweighting strategy: To solve the overfitting problem, LongAlign proposes a strategy that assigns different weights to text-related and text-unrelated parts. The strategy reduces the weight of the text-unrelated part, enhances the model's focus on text content, and improves alignment.
LongAlign's project address
- GitHub repository：https://github.com/luping-liu/LongAlign
- arXiv technical paper：https://arxiv.org/pdf/2410.11817
LongAlign's application scenarios
- Artistic creation: Artists and designers generate images that match detailed descriptions, achieving more precise visual expression in digital art creation.
- Game development: In game design, it helps create images and concept art that match the game's background story or specific scene descriptions.
- Film and entertainment industry: Filmmakers and the entertainment industry generate images that match script or storyboard descriptions, for pre-production or visual effects design.
- Advertising and marketing: Advertising agencies generate images that match advertising copy or marketing campaign descriptions, improving the visual appeal of advertisements.
- Education and training: In the field of education, it generates images that match teaching materials or course content descriptions, enhancing the learning experience.