SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Engram - Conditional Memory Module Jointly Launched by DeepSeek and Peking University

Author: AI小集 · Original source: AI工具集

What Is Engram

Engram is a conditional memory module jointly launched by theDeepSeekteam and Peking University, capable of improving the performance of large language models (LLMs). Engram introduces modern -gram embedding tables to store static knowledge in independent memory, using sparse retrieval and context-aware gating techniques to achieve efficient knowledge lookup. Engram is complementary to Mixture-of-Experts (MoE), and can significantly improve model performance on knowledge-intensive tasks, long-context processing, and reasoning ability, without increasing computation, while also breaking through GPU memory limits and supporting large-scale parameter expansion.

Engram

Engram's Main Features

  • Knowledge Retrieval: Quickly looks up fixed knowledge through static embedding tables, reducing dependence on computational resources.
  • Long-Context Processing: Frees up the capacity of the attention mechanism, allowing it to focus on global context and improving long-text reasoning ability.
  • Decoupling of Computation and Storage: Supports asynchronous data prefetching from host memory, breaking through GPU memory limits and enabling large-scale parameter expansion.
  • Performance Improvement: Without increasing computation, significantly improves model performance on knowledge-intensive tasks, general reasoning, code, and math tasks.

Engram's Technical Principles

  • Tokenizer Compression: Maps original Token IDs to normalized IDs through a precomputed surjective function, reducing vocabulary redundancy and improving -gram coverage efficiency.
  • Multi-Head Hash Retrieval: Uses hashing methods and a multi-head mechanism to map local context into embedding tables, solving the challenge of directly parameterizing -gram combinations.
  • Context-Aware Gating: Uses the current hidden state as a dynamic Query and the retrieved embeddings as Key and Value, suppressing noise through a gating mechanism to ensure semantic consistency.
  • Sparsity Allocation: Under a given parameter budget, reallocates MoE expert parameters to Engram memory, revealing a U-shaped trade-off curve between computation and memory and finding the optimal allocation ratio.
  • System Implementation: Uses Engram's deterministic retrieval characteristics to achieve decoupling of computation and storage, masking data transfer latency through prefetching and overlapping mechanisms, and breaking through GPU memory limits.

Engram's Project Address

  • Technical Paper：https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf

Engram's Application Scenarios

  • Knowledge Question Answering: Engram can quickly retrieve factual information from a knowledge base, improving the response speed and accuracy of question-answering systems.
  • Text Generation: Through efficient knowledge retrieval, Engram can quickly match fixed phrases and patterns in text generation, improving generation efficiency.
  • Long-Text Processing: Engram can optimize long-context modeling and enhance the model's reasoning ability over long texts, making it suitable for long-text analysis in fields such as law and academia.
  • Code Assistance: Engram can quickly retrieve common patterns and structures in code, improving the efficiency of code generation, completion, and understanding.
  • Mathematical Reasoning: Engram can quickly retrieve mathematical formulas and theorems, assisting in solving mathematical problems and improving reasoning efficiency.