What is SVDQuant
SVDQuant is a post-training quantization technique introduced by a research team at MIT. For diffusion models, it quantizes the model's weights and activations to 4 bits, reducing memory usage and accelerating inference. SVDQuant introduces a high-precision low-rank branch to absorb outliers during quantization, achieving 3.5x memory optimization and 8.7x latency reduction on a 16GB 4090 GPU while maintaining image quality. SVDQuant supports the DiT architecture and is compatible with the UNet architecture, and can seamlessly integrate off-the-shelf low-rank adapters (LoRAs) without requantization, providing an effective solution for deploying large diffusion models on resource-constrained devices.
Main Features of SVDQuant
- Quantization Compression: Quantizes the weights and activations of diffusion models to 4 bits, reducing model size and lowering memory usage.
- Accelerated Inference: Quantization reduces computational complexity and improves the model's inference speed on GPUs.
- Low-Rank Branch Absorbs Outliers: Introduces a low-rank branch to handle outliers in quantization, reducing quantization error.
- Kernel Fusion: Designs the Nunchaku inference engine, reducing memory access based on kernel fusion to further improve inference efficiency.
- Supports Multiple Architectures: Compatible with diffusion models of the DiT and UNet architectures.
- LoRA Integration: Seamlessly integrates low-rank adapters (LoRAs) without requantization.
Technical Principles of SVDQuant
- Quantization Processing: Performs 4-bit quantization on the model's weights and activations, posing a challenge to maintaining model performance.
- Outlier Handling: Uses smoothing techniques to transfer outliers in activations to weights, decomposes the weights based on SVD, and decomposes the weights into low-rank components and residuals.
- Low-Rank Branch: Introduces a 16-bit precision low-rank branch to handle outliers in the weights, quantizes the residuals to 4 bits, and reduces quantization difficulty.
- Eckart-Young-Mirsky Theorem: Removes the dominant singular values in the weights, greatly reducing the magnitude and outliers of the weights.
- Inference Engine Nunchaku: Designs an inference engine based on kernels that fuse the low-rank branch and the low-bit branch, reducing memory access and the number of kernel calls, and lowering latency.
SVDQuant Project Address
- Project Website：hanlab.mit.edu/projects/svdquant
- GitHub Repository：https://github.com/mit-han-lab/nunchaku
- arXiv Technical Paper：https://arxiv.org/pdf/2411.05007
- Online Experience Demo：https://svdquant.mit.edu/
Application Scenarios of SVDQuant
- Mobile Devices and Edge Computing: Deploy diffusion models on smartphones, tablets, or other mobile devices to reduce model size and improve inference speed, enabling faster image generation and processing.
- Personal Computers and Workstations: On personal computers or workstations, improve the efficiency of image and video generation, suitable for content creators and professional designers.
- Cloud Computing Platforms: Deploy SVDQuant in cloud services to provide online users with fast image generation services, such as virtual try-on, image editing, and enhancement.
- Interactive Applications: In applications requiring real-time feedback, such as games, virtual reality (VR), and augmented reality (AR), reduce latency and provide a smoother user experience.
- Low-Power Devices: Achieve more energy-efficient model inference in Internet of Things (IoT) devices and other low-power devices.