IndexCache - A Sparse Attention Acceleration Technology Jointly Launched by Tsinghua and Zhipu

What is IndexCache

IndexCache is a sparse attention acceleration technology launched by teams from Tsinghua and Zhipu. It addresses the high computational overhead of the indexer in DeepSeek Sparse Attention (DSA) by reusing indices across layers to reduce redundant computation. IndexCache found that the overlap rate of top-k tokens selected by adjacent layers is as high as 70%-100%, so it divides layers into "full layers" (which compute and cache indices) and "shared layers" (which directly reuse the cache). This method can remove 75% of indexer computation, achieving 1.82x speedup in prefill and 1.48x speedup in decoding in 200K context scenarios, with almost no loss in model performance. It has been validated on 30B parameter models and 744B parameter GLM-5 models.

Main Features of IndexCache

• Cross-layer index reuse: Leveraging the high 70%-100% overlap rate of top-k indices between adjacent layers, shared layers directly reuse the cached indices of full layers, avoiding redundant computation.
• Significantly reduces indexer overhead: It can remove 75% of indexer computation, maintaining model performance with only 1/4 of the indexers.
• Significantly accelerates inference: Achieves 1.82x speedup in prefill and 1.48x speedup in decoding under 200K context, shortening user waiting time.
• Zero additional memory overhead: Reuse is implemented through a conditional branch, without allocating additional GPU memory.
• Provides two deployment schemes: The training-free scheme determines the optimal layer pattern through greedy search, while the training-aware scheme optimizes indexer parameters through a multi-layer distillation loss.
• Production-grade validation: It has been validated on 30B parameter models and the 744B parameter GLM-5, and supports SGLang and vLLM inference frameworks.

Technical Principles of IndexCache

• Discovery of cross-layer index similarity: Through heatmap analysis, the research team found that the top-k token sets output by the indexers of adjacent layers in DSA models have extremely high similarity, with overlap rates generally between 70% and 100%, indicating that a large amount of index computation is redundant.
• Layer role division mechanism: IndexCache divides model layers into two categories: Full Layers retain the original indexer and are responsible for computing and caching the latest top-k indices; Shared Layers no longer run their own indexer and directly reuse the indices cached by the most recent Full Layer for sparse attention computation.
• Dynamic pattern selection strategy: For already-trained models, a greedy search algorithm based on calibration data is used, trying to convert layers into shared layers one by one and evaluating the impact on model output, retaining critical layers as full layers; for training from scratch, a multi-layer distillation loss is introduced, allowing each full-layer indexer to simultaneously learn to serve the needs of multiple subsequent shared layers.
• Inference process optimization: During inference, each layer only adds a simple conditional check, switching between computing new indices and reusing cached indices according to a preset pattern, enabling cross-layer sharing of indexers without modifying the model architecture or adding extra storage.

Key Information and Usage Requirements of IndexCache

• Proposing institution: Jointly developed by Tsinghua University and Zhipu (Z.ai).
• Problem addressed: Solving the computational bottleneck of the indexer in DeepSeek Sparse Attention in long-context scenarios, where it accounts for up to 81% of prefill time at 200K tokens.
• Core principles: Based on the high overlap rate of 70%-100% between top-k indices of adjacent layers, reduce redundant computation through cross-layer reuse.
• Speedup: Retaining only 1/4 of the indexers achieves 1.82x speedup for prefill and 1.48x speedup for decoding.
• Performance loss: Almost no quality loss, and some reasoning tasks even perform better.
• Validated models: Effective on both the 30B-parameter DSA model and the 744B-parameter GLM-5.
• Hardware requirements: Requires NVIDIA GPU (such as H100), but no additional VRAM is needed, reusing the standard DSA memory space.
• Software environment: Supports the SGLang or vLLM frameworks, with ready-made patches that can be directly applied to models such as DeepSeek-V3.2 and GLM-5.
• Training-free solution: Applicable to already-trained DSA models; a small batch of calibration data is needed to run greedy search to determine the optimal layer pattern.

Core advantages of IndexCache