What is PixArt-Σ
PixArt-Σ is a diffusion Transformer architecture (DiT)-based text-to-image model launched by researchers from Huawei Noah's Ark Lab, Dalian University of Technology, and the University of Hong Kong. It is specifically designed to directly generate high-quality images up to 4K resolution from text prompts. This model is built on PixArt-α and, by integrating advanced elements and adopting a weak-to-strong training method, yields a more powerful model that not only improves the fidelity of generated images but also enhances the alignment between images and text prompts. The images generated by PixArt-Σ are comparable in aesthetic quality to current top-tier text-to-image products such as DALL·E 3 and Midjourney V6, and it performs excellently in following text prompts.
PixArt-Σ official website entry
- Official project homepage:https://pixart-alpha.github.io/PixArt-sigma-project/
- GitHub repository:https://github.com/PixArt-alpha/PixArt-sigma(source code and models pending release)
- arXiv research paper:https://arxiv.org/abs/2403.04692
PixArt-Σ features
- 4K resolution image generation: PixArt-Σ can directly generate 3840×2160 high-definition images from text prompts without post-processing or the involvement of other software.
- High-fidelity text-to-image conversion: When generating images, the model can accurately follow the content of the text description, ensuring a high degree of consistency between the image and the text.
- Efficient training: PixArt-Σ adopts a "weak-to-strong training" strategy, significantly improving training efficiency by introducing high-quality data and efficient token compression techniques.
- Smaller model size: Although PixArt-Σ can generate high-resolution images, its number of model parameters is relatively small (0.6B parameters), which makes the model more efficient and easier to deploy.
How PixArt-Σ works
PixArt-Σ works based on theDiT（Diffusion Transformer）architecture, a deep learning method that combines diffusion models and the Transformer architecture and is specifically used to convert text descriptions into corresponding images. The following is a detailed introduction to how PixArt-Σ works:
1. Pretraining stage:
In the pretraining stage, PixArt-Σ uses a large number of text-image pairs to learn the association between text and images. This process usually includes the following steps:
- Text encoding: First, the input text description is converted by a text encoder (such as Flan-T5) into a series of token embedding representations. In PixArt-Σ, to handle more complex descriptions, the token length of the text encoder is extended to about 300 words.
- Conditional feature extraction: Next, these text embeddings, together with the conditional features of the image, are fed into the diffusion model. The conditional feature extractor is usually a pretrained neural network used to extract features related to image generation from the text embeddings.
- Diffusion process: The diffusion model learns the data distribution by simulating a process of gradually adding noise to the data distribution. During training, the model learns how to recover clear images from noisy data.
- Iterative refinement: Through an iterative process, the model gradually learns how to generate high-quality images based on text descriptions. This process involves adjusting model parameters so that the generated images become increasingly close to real images.
2. Weak-to-strong training:
PixArt-Σ adopts a weak-to-strong training strategy, which means the model is first trained on a lower-quality dataset, and then progressively introduces higher-quality datasets and more complex training strategies. This approach helps the model effectively utilize new data and algorithms to enhance performance under limited resources.
3. High-quality training dataset (Internal-Σ):
PixArt-Σ uses a new high-quality dataset, which contains high-resolution images and dense and accurate descriptions. These data provide the model with rich visual information and text information, helping to improve the quality of generated images and the degree of alignment with text descriptions.
4. Efficient token compression and weight initialization:
To generate high-resolution images, PixArt-Σ introduces compression techniques for key and value tokens, as well as a specially designed weight initialization scheme. These techniques enable the model to effectively handle the generation of high-resolution images while reducing the demand for computational resources.
5. Fine-tuning techniques:
In the fine-tuning stage, PixArt-Σ further improves the model's performance by replacing it with a more powerful variational autoencoder (VAE), extending from low resolution to high resolution, and evolving the model from not using KV compression to using KV compression.
6. Generation stage:
In the generation stage, the user inputs a text description, and the model, based on the learned text-image associations, gradually generates an image matching the description through an iterative process. The generated image is adjusted according to the content, style, and details of the text description to ensure that the final output image is both aesthetically pleasing and accurately reflects the intent of the text.