FlashRAG is a Python toolkit for reproducing and developing research on retrieval-augmented generation (RAG), developed by RUC-NLPIR, and its technical paper was accepted to the Resource Track of the 2025 ACM Web Conference (WWW 2025).
Purpose
FlashRAG is designed to help researchers easily reproduce existing SOTA work in the RAG field, or implement custom RAG pipelines and components. It provides a complete framework covering the entire process from corpus processing and index construction to retrieval, generation, and evaluation.
Main Capabilities
- Broad and customizable framework: Includes the core components required for RAG scenarios, such as retriever, reranker, generator, and compressor, supporting flexible assembly of complex pipelines.
- Comprehensive benchmark datasets: Provides 36 preprocessed RAG benchmark datasets for testing and validating RAG model performance.
- Pre-implemented advanced RAG algorithms: Built in 23 advanced RAG algorithms, with reported results, making it easy to reproduce under different settings.
- Reasoning-based methods: Supports 7 reasoning-based methods that combine reasoning capabilities with retrieval, performing better on complex multi-hop tasks.
- Efficient preprocessing stage: Provides various scripts, such as corpus processing, retrieval index construction, document pre-retrieval, etc., simplifying RAG workflow preparation.
- Optimized execution: Uses vLLM and FastChat to accelerate LLM inference, and Faiss to manage vector indexes, improving library efficiency.
- Easy-to-use UI: Provides FlashRAG-UI, which allows quick configuration and experience of implemented RAG baselines through a visual interface, and running evaluation scripts.
Usage
Installation: Can be installed via pip, or cloned and installed from GitHub (requires Python 3.10+). If you need to use vllm, sentence-transformers, or pyserini, optional dependencies can be installed. Because pip installation of faiss has compatibility issues, conda commands must be used for installation.
Corpus construction: Save the corpus as a jsonl file, with one document per line. If using Wikipedia, refer to the documentation to convert it into an indexable format.
Index construction: For dense retrieval methods, use faiss to build the index; for sparse retrieval methods (BM25), use Pyserini or bm25s to build a Lucene inverted index. pooling_method (mean, pooler, or cls) and instruction can be specified. If the retrieval model supports the sentence transformers library, the index can be built directly. Building a bm25 index does not require specifying model_path. Sparse neural retrieval methods (SPLADE) are also supported.
Using ready-made pipelines: Load config (you can input a yaml file or direct variables, with variables taking precedence), load the dataset and initialize the pipeline, and components are loaded automatically. PromptTemplete can be used to specify the input prompt. Execute pipeline.run to get results; the output includes intermediate results and metric scores, and can be saved to a file.
Building custom pipelines: Inherit from BasicPipeline, initialize the required components, and complete the run function. Components can also be used alone and embedded in your own code.
FlashRAG-UI: Provides one-click configuration loading, quick method experience, and efficient benchmark reproduction, supporting visual parameter settings, saving configurations, loading corpora and indexes, switching components and hyperparameters, and reproducing built-in baseline methods.
Applicable Scenarios
Suitable for researchers and developers in the RAG field, for reproducing SOTA work, developing custom RAG pipelines, conducting benchmark testing and evaluation. Also suitable for research work that needs to quickly experience different RAG methods.
Limitations
FlashRAG is still under development, has many issues and room for improvement, and will be continuously updated. Some features may be incomplete, and contributions are welcome.
