SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Kimi Linear - A New Hybrid Linear Attention Architecture Open-Sourced by Moonshot AI

Author: AI小集 · Original source: AI工具集

What is Kimi Linear

Kimi Linear is a new hybrid linear attention architecture launched by Moonshot AI, specifically designed to improve the efficiency and performance of large language models (LLMs) in long-sequence tasks. The core component Kimi Delta Attention (KDA) significantly enhances the model's expressive capability and hardware efficiency through fine-grained channel-wise gating mechanisms and efficient block processing algorithms. Kimi Linear adopts a 3:1 hybrid design of KDA and full attention layers (MLA), greatly reducing KV cache usage (by 75%), and achieving a 6.3x decoding speedup when processing million-level long texts. The Kimi Linear architecture outperforms traditional full attention mechanisms in both short-sequence and long-sequence tasks, and performs excellently in reinforcement learning tasks.

Kimi Linear

Main Features of Kimi Linear

  • Efficient processing of long-sequence tasks: Kimi Linear greatly reduces KV cache usage through its hybrid linear attention architecture (by 75%), achieving a 6.3x throughput improvement in 1M long-text decoding.
  • Precise information management: Kimi Delta Attention (KDA) adopts a channel-wise gating mechanism, enabling the model to precisely and selectively retain key information and forget irrelevant content, enhancing long-sequence processing capability.
  • Enhanced reasoning capability: In reinforcement learning tasks requiring complex reasoning, Kimi Linear performs excellently, with faster training accuracy growth, and its test set performance surpasses that of full attention models.
  • Hardware-friendly design: It adopts efficient block processing algorithms, fully utilizes the Tensor Cores of modern GPUs, achieves high matrix multiplication throughput, and significantly reduces computation time and resource consumption.
  • Adaptability to multiple task scenarios: Kimi Linear performs excellently in both short-sequence and long-sequence tasks, is suitable for various application scenarios such as language understanding, code generation, and mathematical reasoning, and has good generalization ability.

Technical Principles of Kimi Linear

  • Hybrid linear attention architecture: Kimi Linear is based on a 3:1 hybrid design, that is, after every three Kimi Delta Attention (KDA) layers, one full attention layer (MLA) is inserted. The design combines the efficiency of linear attention and the strong expressive capability of full attention, while reducing KV cache usage and improving the model's decoding speed.
  • Kimi Delta Attention（KDA）: KDA is the core module of Kimi Linear and achieves efficient processing through the following mechanisms:
    • Fine-grained gating mechanism: It introduces channel-wise gating, where each feature dimension has an independent forgetting rate, similar to RoPE positional encoding, enhancing the model's perception of positional information.
    • Hardware-efficient block processing algorithm: It adopts a block processing parallel algorithm, reducing computation and improving hardware utilization. The state transition of KDA can be regarded as a special diagonal plus low-rank (DPLR) matrix, reducing computational complexity through a constrained structure.
  • No positional encoding (NoPE): The MLA layers of Kimi Linear do not use any explicit positional encoding (such as RoPE), and completely delegate the encoding of positional information to the KDA layers. The design simplifies the model architecture and enhances the robustness and extrapolation ability of long-text tasks.
  • Combination with Mixture-of-Experts (MoE): Kimi Linear combines Mixture-of-Experts (MoE) technology, expanding the model parameter scale through sparse activation patterns, further improving training and inference efficiency. The total number of model parameters is 48 billion, and only 3 billion parameters are activated per forward pass.

Project Address of Kimi Linear

  • HuggingFace Model Repository：https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct
  • Technical Papers：https://github.com/MoonshotAI/Kimi-Linear/blob/master/tech_report.pdf

Application Scenarios of Kimi Linear

  • Long-Text Generation: Kimi Linear performs excellently when processing million-level long texts, with decoding speed increased by 6.3 times, making it suitable for generating lengthy novels, research reports, and more.
  • Code Generation and Understanding: Its efficient long-sequence processing capability enables it to perform excellently in code generation and understanding tasks, supporting the generation of more complex code logic and long code snippets.
  • Mathematical Reasoning and Problem Solving: In reinforcement learning training for mathematical tasks, Kimi Linear's training accuracy increases faster, and its performance on test sets is better than that of full-attention models, making it suitable for solving complex mathematical problems.
  • Language Understanding and Question Answering: Kimi Linear performs excellently in both short-sequence and long-sequence tasks, is suitable for language understanding, question answering systems, and more, and supports longer context understanding and generation.
  • Multimodal Tasks: Kimi Linear can be used for multimodal tasks, such as image caption generation, video content understanding, and more, supporting longer text descriptions and complex logical reasoning.