What is MHA2MLA
MHA2MLA is a data-efficient fine-tuning method jointly introduced by Fudan University, East China Normal University, Shanghai AI Lab, and other institutions. It is based on the introduction ofDeepSeekmulti-head latent attention mechanism (MLA), optimizes the inference efficiency of any Transformer-based LLM, and reduces inference costs. MHA2MLA is implemented based on two key strategies: first, partial-RoPE, which removes the rotary position embedding (RoPE) dimensions of queries and keys that contribute less to attention scores; second, low-rank approximation, which compresses keys and values based on joint singular value decomposition (SVD) to reduce the memory footprint of the KV cache. MHA2MLA only needs to use 0.3% to 0.6% of the original data for fine-tuning, and while greatly reducing the KV cache (e.g., 92.19%), it can keep the performance loss within a very small range (e.g., LongBench performance drops by only 0.5%).
Main features of MHA2MLA
- Significantly reduces KV cache: Based on low-rank compression technology, it greatly reduces the size of the KV cache (up to 96.87%), lowering memory usage during inference.
- Maintains model performance: Fine-tuning with an extremely small amount of data (0.3% to 0.6% of the original training data) keeps performance loss within a very small range (e.g., LongBench performance drops by only 0.5%).
- Compatible with existing technologies: Used in combination with quantization technologies (such as 4-bit quantization) to further improve inference efficiency.
- Data efficiency: Only a small amount of data is needed to complete the architecture conversion from MHA to MLA, making it suitable for rapid deployment in resource-constrained environments.
Technical principles of MHA2MLA
- Partial-RoPE (partial rotary position embedding): In MHA, rotary position embedding (RoPE) incorporates positional information into query vectors (Q) and key vectors (K) through rotation operations, helping the model capture positional relationships in sequences. By computing the contribution of each dimension to the attention score, the RoPE dimensions with smaller contributions (i.e., the "unimportant" dimensions) are removed, reducing computation and memory usage. This process is called "partial RoPE," which preserves key positional information while freeing up space for low-rank compression.
- Low-Rank Approximation: MLA reduces memory usage based on low-rank joint compression of keys and values (KV). MHA2MLA draws on this idea and performs singular value decomposition (SVD) on the key and value parameter matrices in MHA. The key and value matrices are decomposed into products of low-rank matrices, approximating the original matrices with fewer parameters. To better preserve the interaction information between keys and values, MHA2MLA uses a joint SVD (SVDjoint) strategy to jointly decompose the key and value matrices rather than processing them separately.
Project address of MHA2MLA
- GitHub repository：https://github.com/JT-Ushio/MHA2MLA
- arXiv technical paper：https://arxiv.org/pdf/2502.14837
Application scenarios of MHA2MLA
- Edge device deployment: Reduces model memory usage, making it suitable for resource-constrained smart terminals and IoT devices.
- Large-scale model inference: Reduces the KV cache, improves inference efficiency, and lowers hardware costs and energy consumption.
- Combined with quantization technologies: Combined with quantization technologies to further optimize inference performance, suitable for scenarios such as real-time dialogue and online translation.
- Long text processing: Reduces the memory bottleneck of long text tasks and efficiently handles long document summarization and long-form generation.
- Fast model migration: Only a small amount of data is needed for fine-tuning to quickly convert an MHA model to the MLA architecture, reducing migration costs.