TurboRAG is an open-source project released by MooreThreads on GitHub, containing the implementation code and demo results for the paper "TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text".
Purpose This project targets retrieval-augmented generation (RAG) scenarios, with the goal of accelerating RAG by precomputing KV caches for chunked text, and is used to test TurboRAG's performance versus traditional RAG in time-to-first-token (TTFT).
Main Capabilities The repository provides implementation code and a demo that can compare the TTFT of two methods (TurboRAG and traditional RAG). The project includes some documentation and query examples, located in the documents and questions directories respectively, which can be replaced with the user's own data as needed. During testing, LlamaIndex is used to manage and query indexed documents.
Usage The testing steps described in the repository are divided into two steps:
- Prepare chunk caches: run chunk_cache.py. This script automatically splits documents in the documents directory into chunks of 512 tokens in length, and stores each chunk's KV cache in the chunk_kvcache directory.
- Compare TTFT: run turbo_rag.py to compare the TTFT of the two methods. Note: LlamaIndex is used in the tests and must be installed when setting up the environment.
Results Summary According to the test results it provides, when testing with 10 documents, using KV caches can reduce the average TTFT from about 4.13 seconds to about 0.65 seconds; and as the number of documents increases, the speedup becomes more pronounced.
Applicable Scenarios Suitable for developers and researchers who need to evaluate or study RAG inference acceleration, TTFT optimization, and precomputed KV cache solutions, and can be used for local reproduction of comparative experiments.
Limitations The current material only states that the project is used to test the TTFT of TurboRAG and traditional RAG, and provides the above run steps and example results; it does not provide more detailed functional descriptions, production-grade deployment solutions, or pricing and licensing information. If you use its code or data, the repository indicates that the paper must be cited.
