TurboQuant - A vector quantization algorithm launched by Google

What is TurboQuant

TurboQuant is a vector quantization algorithm launched by Google Research that can compress the KV Cache of large models from 32-bit to 3-bit, achieving a 6x reduction in memory, an 8x increase in inference speed, and zero loss in accuracy. TurboQuant transforms vectors through random rotation into a coordinate system that follows a Beta distribution, combined with 1-bit QJL residual correction, requiring no calibration constants or model fine-tuning, and is plug-and-play. TurboQuant has been verified to support long-context tasks for models such as Gemma and Mistral, providing a key breakthrough for edge device deployment and reducing cloud inference costs.

Main features of TurboQuant

• Extreme compression: Compresses 32-bit floating-point KV Cache to 3-bit, reducing memory usage by more than 6x, while also supporting an extreme compression mode as low as 1-bit.
• Accelerated inference: Through highly vectorized quantized computation, achieves an 8x increase in attention computation speed on H100 GPUs, significantly reducing inference latency.
• Accuracy preservation: In long-context benchmarks such as LongBench and needle-in-a-haystack, the compressed model achieves scores completely consistent with the original model, realizing true zero loss in accuracy.
• Plug-and-play: Adopts a data-independent online quantization strategy, requiring no model retraining, fine-tuning, or calibration for specific datasets, resulting in a low deployment threshold.
• Dual-mode quantization: Provides an MSE optimization mode for minimizing reconstruction error, as well as an inner product optimization mode for providing unbiased attention score estimation, meeting the needs of different application scenarios.
• Multi-scenario applicability: Suitable for KV Cache compression in large models to support ultra-long contexts, and for nearest neighbor search in vector databases, outperforming traditional methods in both recall and indexing speed.

Technical principles of TurboQuant

• Random rotation dimensionality reduction: By applying a random rotation matrix to the input vector, the high-dimensional vector is transformed from a Cartesian coordinate system into a space where each coordinate follows a Beta distribution, making different coordinates nearly independent, thereby allowing optimal scalar quantization to be independently applied to each coordinate without storing data-dependent calibration constants.
• Optimal scalar quantization: Based on the statistical properties of the Beta distribution, uses the Lloyd-Max algorithm to solve the continuous one-dimensional k-means problem, precomputing the optimal quantization codebook for each coordinate to achieve a near-optimal MSE distortion rate.
• Two-stage residual correction: Applies the MSE-optimal quantizer for primary compression, then applies a 1-bit Quantized Johnson-Lindenstrauss transform to the residual vector for correction, eliminating the bias in inner product estimation and achieving unbiased attention computation with low distortion.
• Information-theoretic optimality guarantee: By proving that TurboQuant's distortion rate differs from the Shannon theoretical lower bound by only a constant factor of about 2.7, and is closer to optimal at low bit rates, theoretically validating the algorithm's ultimate performance.

Key information and usage requirements of TurboQuant

• Publisher: Jointly launched by Google Research and Google DeepMind, with the paper published at ICLR 2026.
• Core metrics: KV Cache compressed to 3-bit, memory reduced by 6x, inference speed increased by 8x, zero loss in accuracy.
• Technical combination: Composed of two stages: PolarQuant (random rotation + Beta distribution quantization) and QJL (1-bit residual correction).
• Theoretical guarantee: The distortion rate is within 2.7 times the information-theoretic lower bound, and at 1-bit it differs by only 1.45 times.
• Validated models: Open-source large models such as Gemma and Mistral, tested on 5 long-context benchmarks including LongBench and Needle in a Haystack.
• Community implementations: Multiple third-party implementations have emerged, including PyTorch, MLX, and C/CUDA.
• No training required: No retraining or fine-tuning of the model is needed; it can be applied directly to pretrained models.
• No calibration required: It adopts a data-independent online quantization strategy and does not require offline calibration or preprocessing for a specific dataset.
• Hardware support: AI accelerators that support vectorization (such as GPUs) are needed for best performance; the algorithm itself is not tied to specific hardware.

TurboQuant's core advantages

• Extreme compression ratio: Compresses 32-bit KV Cache to 3-bit, reducing memory usage by more than 6 times, and supports 1-bit extreme compression at minimum, significantly alleviating the VRAM bottleneck in long-context scenarios.
• Zero accuracy loss: Across 5 long-context benchmarks, the compressed model achieves exactly the same scores as the original model, achieving true lossless compression rather than approximately lossless.
• Significant inference acceleration: The highly vectorized algorithm design increases attention computation speed by 8 times, effectively reducing inference latency and improving throughput.
• Plug-and-play deployment: No model retraining, fine-tuning, or data calibration is required, making it ready to use out of the box and greatly lowering the engineering implementation threshold and deployment cost.
• Optimal theoretical performance: The distortion rate differs from the Shannon information-theoretic lower bound by only about a constant factor of 2.7, and the gap is even smaller at low bit widths, approaching the theoretical limit.

How to use TurboQuant

Currently, no official open-source code has been released. You can follow the official Google Research repository or the arXiv paper page for the latest open-source information.

TurboQuant project address

• Project official website：https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/
• arXiv technical paper：https://arxiv.org/pdf/2504.19874

Comparison of TurboQuant with similar competing products

Application scenarios of TurboQuant

• Long-context LLM serving: Compresses the KV Cache by 6x, enabling cloud APIs to support million-level token contexts, significantly reducing compute costs and improving concurrency.
• Consumer-grade GPU deployment: Enables consumer-grade GPUs with 32GB of VRAM to smoothly run long-context tasks for models of 7B and above, breaking the VRAM bottleneck of local deployment.
• Edge device inference: Provides compression solutions for memory-constrained scenarios such as phones and IoT devices, bringing large model capabilities down to end-side devices.
• Vector database retrieval: Replaces traditional Product Quantization, achieving semantic search with higher recall and lower indexing latency in RAG systems.