What is HART
HART (Hybrid Autoregressive Transformer) is an autoregressive visual generation model introduced by a research team at MIT. It can directly generate high-resolution images of 1024×1024 pixels, with quality comparable to diffusion models. HART is based on hybrid Tokenizer technology, which decomposes the continuous latent representation of an autoencoder into discrete tokens and continuous tokens, where the discrete tokens are responsible for capturing the main structure of the image, and the continuous tokens focus on details. HART's lightweight residual diffusion module uses only 37 million parameters, greatly improving computational efficiency. On the MJHQ-30K dataset, HART reduces the reconstruction FID from 2.11 to 0.30 and the generation FID from 7.85 to 5.38, an improvement of 31%, while increasing throughput by 4.5-7.7 times compared with existing diffusion models and reducing MAC by 6.9-13.4 times.
HART's Main Features
- High-Resolution Image Generation: Directly generates high-resolution images of 1024×1024 pixels, meeting the demand for high-quality visual content.
- Improved Image Quality: Based on hybrid Tokenizer technology, HART surpasses traditional autoregressive models in image reconstruction and generation quality, and is comparable to diffusion models.
- Computational Efficiency Optimization: While maintaining high image quality, it significantly improves computational efficiency and reduces training costs and inference latency.
- Autoregressive Modeling: Based on an autoregressive method, it generates images step by step, supporting finer control over the generation process.
HART's Technical Principles
- Hybrid Tokenizer: The core of HART is the hybrid Tokenizer, which decomposes the continuous latent representation of an autoencoder into discrete tokens and continuous tokens. The discrete tokens are responsible for capturing the main structure of the image, and the continuous tokens focus on details.
- Discrete Autoregressive Model: The discrete part is modeled by a resolution-scalable discrete autoregressive model, supporting the model in generating images at different resolutions.
- Lightweight Residual Diffusion Module: The continuous part is learned by a lightweight residual diffusion module, which has only 37 million parameters and helps improve the model's efficiency.
- Balance Between Efficiency and Performance: HART outperforms existing diffusion models on FID and CLIP scores, increases throughput by 4.5-7.7 times, and reduces MAC by 6.9-13.4 times, achieving a good balance between efficiency and performance.
- Autoregressive Generation: HART is based on an autoregressive method, generating images step by step, with each step based on the output of the previous step, supporting the model in progressively refining image details during the generation process.
HART's Project Address
- Project Official Website：hanlab.mit.edu/projects/hart
- GitHub Repository：https://github.com/mit-han-lab/hart
- arXiv Technical Paper：https://arxiv.org/pdf/2410.10812
- Online Experience Demo：https://hart.mit.edu/
HART's Application Scenarios
- Digital Art Creation: Artists and designers generate high-quality digital artworks, including illustrations, concept art, and visual effects.
- Game Development: In game design, generate game assets such as high-resolution images of characters, environments, and props.
- Film and Video Production: Generate movie posters, concept art, or serve as backgrounds and special effects for video content.
- Advertising and Marketing: Marketing teams quickly generate attractive advertising images and marketing materials.
- Social Media Content: Users generate personalized images and visual content for social media platforms.