langchain-ChatGLM

Langchain-Chatchat (formerly langchain-ChatGLM) is an open-source, offline-deployable knowledge base Q&A application, implemented based on the Langchain framework and large language models such as ChatGLM, Qwen, and Llama. The project aims to establish a RAG (Retrieval-Augmented Generation) and Agent application solution that is friendly to Chinese scenarios and open-source models and can run completely offline.

[Purpose] The project is mainly used to build a Q&A system based on a local knowledge base. Users can import document materials into the knowledge base, and the system uses retrieval-augmented generation to answer questions related to the document content with a large language model. In addition, version 0.3.x also supports Agent functionality, which can be used to handle more complex tasks.

[Main Capabilities]

1. Supports mainstream open-source LLMs, Embedding models, and vector databases, enabling fully offline private deployment using only open-source models.
2. Supports integration with model inference frameworks such as Xinference, Ollama, LocalAI, FastChat, and One API.
3. Supports recently open-sourced large language models and Embedding models such as GLM-4-Chat, Qwen2-Instruct, and Llama3. Users need to start the model deployment framework themselves and integrate the models by modifying the configuration information.
4. Supports online API integration, including commonly used online APIs such as OpenAI ChatGPT, Azure OpenAI API, Anthropic Claude, Zhipu Qingyan, and Baichuan.
5. Provides an API calling service based on FastAPI, as well as a WebUI operation interface based on Streamlit.
6. Since version 0.3.0, an installation method in the form of a Python library has been provided.
7. Uses the FAISS knowledge base by default, and the configuration can be modified to connect to other types of knowledge bases.
8. Supports use in Python 3.8-3.11 environments, and has been tested on Windows, macOS, and Linux operating systems.
9. Because version 0.3.0 supports integration with different model deployment frameworks, it can be used under different hardware conditions such as CPU, GPU, NPU, and MPS.

[Usage]

1. Installation and deployment: Can be installed via pip, source code installation deployment/development deployment, or Docker deployment. Since version 0.3.0, installation in the form of a Python library has been provided.Installation.
2. Model integration: Starting from version 0.3.0, the project no longer directly loads models based on the local model path entered by the user, but instead supports integration with common major model inference frameworks on the market. Before starting the project, you need to first run the model inference framework and load the required models. It is recommended to place Langchain-Chatchat and the model deployment framework in different Python virtual environments to avoid dependency conflicts.
3. Configuration: Starting from version 0.3.1, local yaml files are used for configuration. Users can directly view and modify them, and the server updates automatically without restarting. The main configuration files include model_settings.yaml (configure the default LLM, Embedding model, and model platform information), basic_settings.yaml (configure the knowledge base path, database path, listening address, etc.), and kb_settings.yaml (configure the knowledge base type).
4. Initialize the knowledge base: Before initializing the knowledge base, make sure that the model inference framework and the corresponding embedding model have been started, and that the model integration configuration has been completed. You can use the chatchat kb command to perform knowledge base operations.
5. Start: After configuration is complete, you can start the service and operate it through the WebUI or API. The default listening address is 127.0.0.1. If you need to access it through another IP, you need to change the listening address to 0.0.0.0.

[Applicable scenarios] Suitable for scenarios that require question answering based on a local knowledge base, especially scenarios with requirements for data privacy and offline deployment, such as enterprise internal knowledge bases, document question answering systems, etc. The project is friendly to Chinese scenarios and open-source models, and is suitable for Chinese users.

[Limitations and notes]

1. The project does not involve fine-tuning or training processes, but fine-tuning or training can be used to optimize the project's performance.
2. The structure of version 0.3.x has changed significantly. When upgrading, it is recommended to redeploy according to the documentation and back up important data in advance.
3. Models need to be integrated by users themselves after starting the model deployment framework and modifying the configuration.
4. Some features may require additional dependency libraries to be installed. For example, when used with the Xinference framework, the corresponding Python dependencies need to be installed.
5. If you encounter the problem of getting stuck during knowledge base initialization, you can try executing the relevant command and then recreating the knowledge base.
6. There are dedicated instructions for database conversation configuration, which need to be referred to separately.
7. The project code follows the Apache-2.0 license.

Project history: In April 2023, version 0.1.0 was released, supporting local knowledge base question answering based on ChatGLM-6B model; in August 2023, it was renamed Langchain-Chatchat and version 0.2.0 was released, using fastchat as the model loading solution; in October 2023, version 0.2.5 was released, introducing Agent content; in December 2023, it received more than 20K stars; in June 2024, version 0.3.0 was released, bringing a completely new project architecture.