PixArt-Σ is a diffusion Transformer project aimed at 4K text-to-image generation, with the paper title "PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation." This repository contains PyTorch model definitions, pretrained weights, and inference/sampling code for exploring the "weak-to-strong training" method. The project was jointly completed by researchers from Huawei Noah's Ark Lab, Dalian University of Technology, the University of Hong Kong, the Hong Kong University of Science and Technology, and other institutions. The authors include Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li.
In terms of use, PixArt-Σ is mainly used for text-to-image generation, especially for 4K resolution scenarios. It continues the work of the PixArt-α project and attempts to keep the codebase as simple as possible for the PixArt community to use and contribute to.
Main capabilities and released content:
- Model checkpoints: Checkpoints at different resolutions such as 256px, 512px, 1024px, and 2K have been released.
- Inference and sampling: The repository provides inference/sampling code and supports quick launch via Gradio.
- diffusers integration: diffusers has been supported since April 24, 2024. diffusers can be upgraded to use PixArtSigmaPipeline; for versions diffusers<0.28.0, the official team provides script instructions.
- LoRA: LoRA code has been released.
- One Step Generation (DMD): The PixArt-α-DMD one-step sampler demo code and 512px checkpoint have been released, and the training code for the DMD one-step generator has also been released.
- Training code: Training and inference code and toy data have been released. The data supports training without prior feature extraction.
- Online Demo: PixArt-Σ Online Demo is provided.
- Other features: feature extraction (optional), data reformatting (optional).
In terms of usage, the dependency environment is Python >= 3.9 (Anaconda or Miniconda recommended), PyTorch >= 2.0.1+cu11.7. The training process includes: downloading the toy dataset, downloading pretrained checkpoints, and selecting the required configuration file from the configuration file directory. Quick start can be done through Gradio: first install dependencies, and download the checkpoint files to the output/pretrained_models folder. diffusers integration requires upgrading diffusers to enable PixArtSigmaPipeline. In addition, the repository also provides instructions for converting .pth checkpoints to the diffusers version and supports downloading models directly from Hugging Face.
Applicable scenarios include: text-to-image generation research, 4K image generation experiments, diffusion Transformer training and fine-tuning, LoRA training and inference, ControlNet-related development, etc.
Limitations and information not provided: As of the information shown, LCM, ControlNet, ComfyUI, etc. are marked as "coming soon" and have not yet been released. Content such as "KV Compress..." in the model zoo is also marked as planned for release. Details regarding charging, pricing, platform restrictions, specific evaluation metrics for generation quality, and minimum hardware requirements are not yet provided in the information.
