XRAG

XRAG (full name eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation) is a benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems, designed to systematically evaluate the performance of the foundational component modules of RAG systems. The project is hosted on GitHub (https://github.com/DocAILab/XRAG），由 maintained by DocAILab, and the paper has been accepted by ICDE 2026.

[Purpose] XRAG is mainly used in research and development scenarios to benchmark and evaluate the retrieval and generation components of RAG pipelines, helping developers and researchers compare the effects of different retrieval strategies, embedding models, and LLMs, and analyze retrieval quality, response faithfulness, and answer correctness.

[Main Capabilities]

1. Comprehensive evaluation framework: supports multi-dimensional evaluation, including LLM-based evaluation, deep evaluation, and traditional metrics; built-in LlamaIndex, DeepEval, and custom evaluation models.
2. Rich evaluation metrics: traditional metrics such as F1, EM, MRR, Hit@K, MAP, NDCG; LLM metrics such as Faithfulness, Relevancy, Correctness; deep evaluation metrics such as Contextual Precision/Recall, Hallucination, Bias.
3. Flexible architecture: modular design, with retrievers, embedding models, and LLMs pluggable; supports vector retrieval, BM25, hybrid retrieval, tree retrieval, keyword retrieval, document summary retrieval, and custom retrieval strategies.
4. Multi-LLM support: seamlessly integrates OpenAI models, supports local models such as Qwen and LLaMA, supports using models with 30B or more parameters via the OpenAI API or Ollama; model parameters and API settings are configurable.
5. Novel evaluation metrics: including Semantic Perplexity (SePer), Entropy, and Semantic Entropy.
6. Agent workflows: supports Adaptive Retrieval and Multi-step Approach; orchestrators such as self-rag, adaptive-rag, and SIM-rag have been added.
7. Text splitters: including SemanticSplitterNodeParser, SentenceSplitterNodeParser, and SentenceWindowNodeParser.
8. Embedding models: supports PGE, GTE, and M3E embeddings.
9. User interface: provides a feature-rich command-line interface and a W for interactive evaluation and visualizationebUI, and generate detailed evaluation reports and visualization results.

[Usage] Installation requirements: Python 3.11 or higher. It is recommended to create a virtual environment via conda, or you can directly use pip to install XRAG. Configuration: Modify the config.toml file to set the required configuration. Main commands of the command-line tool xrag-cli:

• run: Run the benchmark test, supporting --override key=value to override the configuration, for example xrag-cli run [--override key = value ...].
• webui: Start the web user interface, xrag-cli webui.
• ver: Display the current version, xrag-cli version.
• help: Display help information, xrag-cli help.
• generate: Generate QA pairs from a folder, xrag-cli generate -i <input_file> -o <output_file> -n <num_questions> -s <sentence_length>.
• api: Start the API server for the XRAG service, xrag-cli api [--host <host>] [--port <port>] [--json_path <json_path>] [--dataset_folder <dataset_folder>], the default host is 0.0.0.0, and the default port is 8000.

WebUI workflow:

1. Dataset upload and configuration: Supports benchmark datasets (HotpotQA, DropQA, NaturalQA) and custom datasets, with automatic format conversion and preprocessing.
2. Index construction and configuration: Configure API keys, parameter settings, vector database index construction, and chunk size optimization.
3. RAG strategy configuration: Define pre-retrieval methods, retriever selection, post-processor configuration, and custom prompt template creation.
4. Interactive testing: Real-time query testing, retrieval result viewing, response generation review, and performance analysis.
5. Comprehensive evaluation.

API service: After startup, interaction is possible through HTTP requests. Send a POST request to /query to obtain document-based answers; check the server status via GET /health. Supports custom JSON datasets and folder-based documents: --json_path is used for JSON-format QA datasets, and --dataset_folder is used for document folders; the two should not be set at the same time.

[Applicable scenarios] Suitable for RAG system research, retrieval strategy comparison, embedding model and LLM selection evaluation, ablation experiments on RAG pipeline components, and development and research scenarios that require generating QA pairs or conducting retrieval quality and generation quality analysis.

[Limitations] Installation requires Python 3.11 or higher. During use, issues such as dependency conflicts, mismatched configuration key names, and data type errors when overriding configurations may be encountered. It is necessary to ensure that the versions in requirements.txt are correct and to use a virtual environment. --json_path and --dataset_folder cannot be set at the same time. Information such as the project's specific fees, pricing, and platform support scope has not yet been provided.

[Update log] 2026-10-08 Added XRAG WebUI; 2026-02-24 The paper was accepted by ICDE 2026; 2025-11-18 Added orchestrators SIM-rag; 2025-11-05 Added self-rag and adaptive-rag; 2025-07-18 Added text splitter; 2025-06-23 Updated configuration file; 2025-01-09 Added API support; 2025-01-06 Added ollama LLM support; 2025-01-05 Added the generate command; 2024-12-23 Released XRAG documentation; 2024-12-20 Released XRAG.

[Feedback and support] You can submit an issue through the GitHub repository, or contact us by sending an email to luoyangyifei@buaa.edu.cn.