RedKnot - Xiaohongshu's open-source long-context inference acceleration engine

What is RedKnot

RedKnot is Xiaohongshu's open-source long-context LLM inference acceleration engine. It decouples the KV Cache along the attention head dimension, offline classifying heads into global heads that require global recomputation and local heads that can be locally reused. Combined with SegPagedAttention, it implements per-head paged storage and a fused variable-length attention kernel, avoiding the kernel penalty caused by attn_mask. RedKnot also introduces sparse FFN, performing dense computation on tokens with high attention scores to address the short-context FFN bottleneck.

RedKnot's main features

• Head classification sparsity: Offline classifies each pair into global heads that require global recomputation and local heads that can be locally reused, remaining stable across requests without online overhead.
• Position-independent KV reuse: Supports reusing precomputed KV Cache when the same document segment appears at non-prefix positions, breaking the limitations of traditional prefix matching.
• Sparse FFN acceleration: Performs dense FFN computation on the top-k tokens with the highest attention scores, while the remaining tokens take the residual identity path, directly reducing the short-context FFN bottleneck.
• SegPagedAttention storage: Changes the KV Cache from a dense tensor to segmented paged storage, physically retaining only the tokens each head truly needs.
• Elastic sparse recovery: Shallow layers adopt conservative local attention plus dense FFN to protect the early residual stream, while deep layers enable global head recomputation and sparse FFN, balancing accuracy and efficiency.
• Architecture-agnostic runtime: Through the four adapter interfaces PROFILE, BUILDSTATE, SELECTVISIBLESTATE, and EXECUTE, it uniformly supports different attention architectures such as GQA, MoE, and MLA.

Follow WeChat and reply "open source" to join theAI open-source project communication group

RedKnot's technical principles

• Per-head decoupled KV Cache: Traditional systems treat the KV Cache as a dense [B,H,L,D] tensor. RedKnot found that the effective context range and importance of different attention heads are highly differentiated, so it splits the cache system along the head dimension.
• Head-level sparsity replaces token-level sparsity: Token-level recovery requires taking the union of important tokens from each head, causing expansion. RedKnot switches to per-head recovery, recomputing only a few global heads and directly reusing local heads, avoiding the accuracy-latency dilemma.
• Orthogonal optimization of FFN and attention: In short contexts (2–8K), FFN accounts for 57–62% of TTFT, which attention optimization cannot touch; RedKnot uses attention signals to select important tokens for FFN, forming multiplicative stacked gains with KV sparsity.
• Paged storage eliminates mask penalty: Dense layout plus attn_mask disables the FlashAttention fast path, bringing a 4.9–7.6× kernel penalty. SegPagedAttention pages by head and works with a fused varlen kernel, constructing no mask throughout.
• Layered elastic strategy: In shallow layers of the model, local heads account for a high proportion and semantic selectivity is weak, so conservative recovery prevents error propagation; in deep layers, the proportion of global heads rises and attention becomes more concentrated, so the sparse strategy yields the greatest benefit with the smallest accuracy loss.

How to use RedKnot

• Offline profiling classification: Run against the target model PROFILE interface, and through the needle-in-a-haystack test, offline classify each (layer, head) pair as a global head or local head, generating a stable Head Class Map for zero-overhead reuse in subsequent requests.
• Prebuilt reusable state: Use the BUILDSTATE interface to precompute commonly used document fragments into KV Cache, and store them by head dimension in pages in the Global / Local KV Pool, enabling position-independent offline preconstruction.
• Dynamic state selection: Upon receiving an online request, use the SELECTVISIBLESTATE interface to select, based on query semantics and the Head Class Map, the local-head KV to be reused and the global-head range to be recomputed from the cache pool.
• Fused inference execution: Call the EXECUTE interface; global heads undergo full attention recomputation and are written to the Online KV Cache, local heads directly reuse paged KV for local attention, and low-score tokens skip FFN computation via the residual path.
• Service deployment integration: Pull the source code implemented based on SGLang from the GitHub repository https://github.com/rednote-machine-learning/RedKnot , and integrate it into the existing inference service stack according to the four major adapter interface specifications for deployment.

RedKnot's core advantages

• Head-wise decoupled granularity alignment: Break through the traditional token-level dense KV Cache abstraction, unifying storage, computation, and recovery granularity to the attention head dimension, matching the workload's true head-wise sparse structure.
• Offline profiling with zero online overhead: The classification of global heads and local heads is highly stable across requests, requiring only a one-time offline profiling; at online runtime, it is directly applied via table lookup without adding any inference latency.
• Head-level recovery replaces token-level recovery: Only about 12–15% of global heads are recomputed, while 85–88% of local heads are directly reused, avoiding runaway recomputation caused by token-level union expansion while eliminating cascading error propagation.
• Orthogonal acceleration via sparse FFN: Execute dense FFN for the top-k tokens with high attention scores, and route the rest through the residual path, directly reducing the FFN bottleneck that accounts for 57–62% of TTFT in short contexts, creating multiplicative additive gains with attention optimization.

RedKnot's project address

• GitHub repository：https://github.com/rednote-machine-learning/RedKnot
• arXiv technical paper：https://arxiv.org/pdf/2606.06256

Comparison of RedKnot with similar competitors

Application scenarios of RedKnot

• RAG long-document question answering: Concatenate tens of thousands of retrieved chunks into the prompt, and through position-independent KV reuse and head-level sparse recovery, reduce long-text prefill latency from tens of seconds to a few seconds.
• Coding Agent multi-turn tool calling: Call tools dozens of times consecutively and accumulate historical context, using sparse FFN to directly cut the FFN bottleneck that accounts for more than half of TTFT on short segments (2–8K).
• Long-session memory systems: Bring user memory, tool outputs, and historical state uniformly into a long context, and through head-level reuse increase single-GPU concurrency from 4-way to more than 30-way.
• Multi-Agent collaboration frameworks: Multiple Agents dynamically exchange and reorder context segments; position-independent KV reuse breaks the limitation of "must prefix-match" and avoids repeated prefill.
• Real-time streaming long-text generation: Local heads directly reuse recent paged KV, while global heads recompute on demand, maintaining low first-token latency and high generation stability even under a 128K context.