CAG

CAG (Cache-Augmented Generation) is a generation paradigm proposed to replace real-time retrieval with caching, intended as a simple and efficient alternative to RAG (Retrieval-Augmented Generation).

Purpose and Background RAG enhances language models by integrating external knowledge sources, but it introduces several challenges: retrieval latency (delay caused by real-time retrieval steps), retrieval errors (inaccuracies in selecting relevant documents), and system complexity (increased architecture and maintenance overhead). CAG addresses these limitations by bypassing real-time retrieval.

Main Principle CAG leverages the extended context windows of modern large language models (LLMs) to preload all relevant resources into the model context and cache their runtime parameters. During inference, the preloaded KV cache enables the model to generate answers directly without retrieval.

Advantages of CAG

• Reduced latency: eliminates real-time retrieval, enabling faster inference.
• Improved reliability: reduces retrieval errors while maintaining context relevance.
• Simplified design: provides a streamlined alternative without retrieval, achieving comparable or better results with lower complexity.

Limitations of CAG

• Limited knowledge scale: CAG requires the entire knowledge source to fit within the context window, making it less suitable for tasks involving extremely large-scale datasets.
• Context length constraints: LLM performance may degrade with very long contexts.

Papers and Research Progress The related paper will be published as a short paper at ACM Web Conference 2025, studying the relationship between model performance and context length, and providing an analysis of scenarios where CAG performs well. The limitations of CAG are rapidly being improved with LLMs that have longer context windows and enhanced ability to extract relevant information from extended inputs. As these models continue to evolve, CAG is expected to handle increasingly complex applications, becoming a practical and scalable alternative to traditional RAG.

Usage and Project Files

• Download the required squad and hotpotqa datasets via the curl script.
• Create a .env file according to .env.template and fill in the required keys.
• rag.py is used for RAG experiments.
• kvcache.py is used for CAG experiments.

kvcache.py Parameter Usage --kvcache : "file" --dataset : "hotpotqa-train" or "squad-train" --similarity "bertscore" --modelname : "meta-llama/Llama-3.1-8B-Instruct" --maxKnowledge : "", int, how many documents to select from the dataset, see Note for details --maxParagraph : 100 --maxQuestion int, maximum number of questions, see Note for details --randomSeed : "", int, random seed number --output : "", str, output file path string --usePrompt, add this parameter if you do not want to use CAG knowledge cache acceleration kvcache.py example (the material is labeled "Example -- kvcache.py", no specific command content provided).

rag.py Parameter Usage --index : "openai" or "bm25" --topk : int, topk for retrieval similarity rag.py example (the material is labeled "Example -- rag.py", no specific command content provided).

Dataset Scale and Token Count Description "squad-train" dataset:

• When k = 3, tokens = 21,000
• When k = 4, tokens = 32,000
• When k = 7, tokens = 50,000 "hotpotqa-train" dataset:
• All k = 7405 articles, tokens = 10,038,084
• When k = 1, tokens = 1,400
• When k = 16, tokens = 22,400
• When k = 24, tokens = 33,667
• When k = 32, tokens = 44,800
• When k = 48, tokens = 64,000
• When k = 64, tokens = 85,000
• When k = 80, tokens = 106,000 When using the "squad-train" dataset, 1 piece of knowledge corresponds to an average of 150 questions; when using the "hotpotqa-train" dataset, 1 piece of knowledge corresponds to 1 question. Since "hotpotqa-trIn the "ain" dataset, 1 document has only 1 question, which may not be sufficient for large-scale evaluation; multiple evaluations may be a relatively better approach.

Docker Build and Run

• Build the Docker image and run the corresponding command (the specific command content is not provided in the materials).
• When running the container, GPU users should run the corresponding command (the specific command content is not provided in the materials).
• If the contents of the .env file are empty at build time, an error similar to the one shown below will occur, so be sure to populate the .env file before building the Docker image (the specific error example is not provided in the materials).
• Note that the CMD instruction in the Dockerfile runs the kvcache.py script by default. If you want to execute other scripts such as rag.py, you can override it in the docker run command (the specific command content is not provided in the materials).

Acknowledgments This work was partially supported by the National Science and Technology Council (NSTC) of Taiwan (Grant No. 112-2221-E-001-016-MY3), Academia Sinica (Grant No. 236d-1120205), and the National Center for High-performance Computing (NCHC) of the National Applied Research Laboratories and the NSTC under the "Trustworthy AI Dialog Engine, TAIDE" project for the promotion of trustworthy generative AI large language models and literacy cultivation in Taiwan for the year 113. The authors thank Discover AI and the many people who introduced, shared, and discussed this work, contributing to its broader visibility and impact.