SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Delta-CoMe - A New Incremental Compression Algorithm Open-Sourced by Tsinghua University in Collaboration with OpenBMB and Other Universities

Author: AI小集 · Original source: AI工具集

What is Delta-CoMe

Delta-CoMe is a new incremental compression algorithm proposed by the Tsinghua University NLP Lab in collaboration with the OpenBMB open-source community, Peking University, and Shanghai University of Finance and Economics. A single 80G A100 GPU can easily load up to 50 7B models, saving about 8 times the GPU memory, while model performance is almost comparable to the fine-tuned model before compression. Delta-CoMe is based on combining low-rank decomposition and low-bit quantization techniques, using the low-rank property of model parameter increments (Delta) to achieve mixed-precision compression. This method can greatly reduce storage and inference costs while keeping model performance almost lossless, and performs excellently especially when handling complex tasks such as mathematics, code, and multimodal tasks.

Delta-CoMe

Delta-CoMe's Main Features

  • Model Compression: Based on mixed-precision compression technology, it significantly reduces the storage and memory requirements of large language models (LLMs), making it possible to deploy more models on limited hardware resources.
  • Performance Retention: While compressing, it maintains model performance, especially in complex tasks such as mathematical problem solving, code generation, and multimodal tasks, ensuring that the performance of the compressed model is close to that of the uncompressed fine-tuned model.
  • Multi-Task Processing: It supports the simultaneous deployment of multiple models with different capabilities, is suitable for multi-tenant and multi-task processing scenarios, and improves the flexibility and efficiency of model deployment.
  • Inference Speed Improvement: Based on the implementation of Triton kernel operators, inference speed is nearly 3 times faster than the PyTorch implementation, further optimizing the model's operating efficiency.

Delta-CoMe's Technical Principles

  • Low-Rank Decomposition: It uses singular value decomposition (SVD) to perform low-rank decomposition on model parameter increments (Delta), finding that Delta parameters have a low-rank property, that is, most changes are concentrated in a few principal components.
  • Mixed-Precision Quantization: Based on the results of low-rank decomposition, a mixed-precision quantization method is used to assign different bit widths to different singular vectors. Singular vectors corresponding to larger singular values are represented with higher precision, while singular vectors corresponding to smaller singular values are represented with lower precision, reducing storage requirements.
  • Utilization of Long-Tail Distribution: Delta-CoMe observes that the singular values of Delta parameters exhibit a long-tail distribution, that is, most singular values are small, and only a few singular values are large. Therefore, more aggressive compression is performed on singular vectors corresponding to small singular values, while higher precision is retained for singular vectors corresponding to large singular values.
  • Compatibility and Generalization: The Delta-CoMe method is applicable to specific models or tasks, and also has good generalization ability. It is compatible with different backbone models (such as Llama-2, Llama-3, and Mistral) and maintains good performance on a variety of tasks.
  • Hardware Optimization: To further improve inference speed, Delta-CoMe implements Triton kernel operators for mixed-precision quantization, making deployment on hardware more efficient.

Delta-CoMe's Project Address

Delta-CoMe's Application Scenarios

  • Multi-Tenant Services: In cloud computing environments, it supports providing dedicated models for each customer on limited hardware resources, saving GPU memory and computing resources.
  • Multi-Task Processing: In scenarios that require handling multiple different tasks simultaneously, it effectively compresses and deploys models for different tasks.
  • On-Device Deployment: On resource-constrained edge computing devices, reduce the model's storage and memory footprint.
  • Model fine-tuning service: For models that require frequent fine-tuning to adapt to new data or new tasks, compress the increment of the fine-tuned model to reduce storage and inference costs.
  • Academic research and development: Researchers and developers compress and deploy large language models for experimentation and development, without being limited by hardware resources.