SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

REFRAG - An Efficient Decoding Framework Introduced by Meta

Author: AI小集 · Original source: AI工具集

What Is REFRAG

REFRAG is an efficient decoding framework introduced by Meta Superintelligence Labs for retrieval-augmented generation (RAG) tasks. It optimizes the way large language models (LLMs) process external knowledge through a process of "Compress, Sense, Expand." REFRAG splits retrieved long texts into multiple "chunks," generates a compact vector representation for each "chunk," shortens the input sequence length, and reduces computation. The model uses a reinforcement learning policy network to intelligently determine key information and retain the original text of important text chunks. While significantly improving time to first token (up to 30x speedup), the framework maintains performance comparable to full-context models, effectively solving the efficiency problem of large models when processing long contexts.

REFRAG

REFRAG's Main Features

  • Significantly reduces time to first token (TTFT): By optimizing the decoding process, REFRAG speeds up time to first token by up to 30x, significantly improving the system's real-time interaction performance.
  • Maintains or improves the quality of generated content: While accelerating, REFRAG shows no performance loss compared with baseline models using the full context in terms of perplexity and accuracy across various downstream tasks, and performs better on some tasks.
  • Extends the context window: Through compression technology, REFRAG enables the model to process more contextual information under the same computational budget, effectively expanding the context window by 16x, which helps improve the model's performance on tasks that require long contextual information.
  • Adapts to multiple application scenarios: REFRAG is suitable for RAG tasks and can be applied to other tasks that require processing long contextual information, such as multi-turn dialogue and long document summarization, giving it broad applicability.

REFRAG's Technical Principles

  • Compress: Split the retrieved long reference materials into multiple "chunks," generate a compact vector representation called a "chunk embedding" for each "chunk," shorten the input sequence length, reduce subsequent computation, and avoid repeated encoding computation.
  • Sense: By training a policy network based on reinforcement learning (RL), analyze all "chunk embeddings" and the user question, and determine which text chunks contain the most core information and need to be presented to the LLM in the form of original text, ensuring that key information is not lost due to compression.
  • Expand: The final input to the main LLM is a hybrid sequence containing the "chunk embeddings" of most of the context and a small number of "original text chunks" judged to be critical. The LLM generates answers based on the optimized input materials, retains key information, and minimizes computational load.
  • Leverages the sparsity of the attention mechanism: REFRAG observes that in RAG tasks the model's attention mechanism exhibits a "block-diagonal" sparse pattern, that is, the model's attention is mainly concentrated within individual documents and on the associations between each document and the user question. REFRAG reduces unnecessary computation and improves efficiency by selectively compressing and expanding the context.

REFRAG's Project Address

  • arXiv Technical Paper：https://arxiv.org/pdf/2509.01092

REFRAG's Application Scenarios

  • Retrieval-augmented generation (RAG) tasks: By optimizing the decoding process, it significantly improves time to first token, making it suitable for scenarios that require quickly generating accurate answers, such as intelligent customer service and online question-answering systems.
  • Multi-turn dialogue systems: In multi-turn conversations, efficiently handle long conversation histories, maintain conversation coherence and accuracy, and improve user experience.
  • Long Document Summarization: REFRAG can effectively process long documents and generate high-quality summaries, suitable for automatic summary generation of long texts such as news and academic papers.
  • Knowledge Graph Question Answering: Combine knowledge graphs to quickly retrieve relevant knowledge and generate accurate answers, suitable for intelligent question answering systems driven by knowledge graphs.
  • Content Creation Assistance: In content creation scenarios, quickly generate creative text, help authors quickly conceive and write articles, stories, etc., and improve creation efficiency.