localGPT

LocalGPT is a fully private, locally deployed document intelligence platform whose goal is to let users ask questions, summarize, and mine insights from their own files using advanced AI models without transferring data off their machine. It is positioned as a tool that goes beyond traditional RAG (retrieval-augmented generation), with a modular, lightweight architecture at its core.

Purpose

LocalGPT is used to interact with documents on this device: ask questions, summarize, and discover information from files. All processing is done locally, data does not leave the user's machine, and it emphasizes 100% privacy.

Main capabilities

• Extreme privacy: Data remains on the local computer and is not transmitted externally.
• Flexible model support: Generation models can be freely swapped via Ollama.
• Diverse embeddings: Supports HuggingFace embedding models (harrier-oss-v1, Qwen3-Embedding series) or any Ollama embedding tag.
• LLM reuse: Once a model is downloaded, it can be reused without repeated downloads.
• API: Provides a REST gateway on port 8000 and a RAG API on port 8001, making it easy to build your own applications.
• Device adaptation: Embedding and reranking automatically choose between CUDA, Apple MPS, and CPU (trying CUDA, Apple MPS, then CPU in order).

For document processing: supports PDF, DOCX, HTML/HTM, Markdown, and TXT, parsed by Docling; for PDFs without a text layer, it uses Docling's OCR process, and the OCR engine is selected from installed engines (on macOS, OcrMac is preferred, followed by EasyOCR, RapidOCR, tesserocr, or the tesseract CLI). It also provides context augmentation (chunk-level context generated by a small LLM), Late Chunking that is off by default (document-level secondary embedding, stored in a companion table), and document overviews (a short summary of each document, used for routing).

AI chat capabilities include: natural language questioning, source attribution (answers include the text chunks they are based on), intelligent routing (choosing per query between RAG and direct LLM answers), query decomposition (splitting complex questions into sub-questions, retrieving separately, merging candidates, then performing one rerank and one synthesis), Reciprocal Rank Fusion (vectorFused with full-text hits using RRF, no weight tuning required), reranking (cross-encoder reranking of the fused candidate set, enabled by default, selected by calibrated score), sentence pruning (optional Provence pruning, discarding irrelevant sentences from each chunk), semantic caching (session-scoped TTL cache with a similarity threshold of 0.98), and answer verification, disabled by default (a second pass appends [Confidence: N%] after the answer).

On the developer-friendliness side: every UI action is a documented HTTP call; each pipeline stage is exposed as a streaming stage via Server-Sent Events; models, chunk sizes, retrieval modes, etc. are configurable per request; rag_system/main.py holds all defaults and can be overridden by environment variables. The interface provides a clean, responsive Web UI, session management that organizes conversations by topic, index management, and real-time progress for the retrieval/reranking/synthesis stages.

Usage

Runtime requirements: Python 3.10+ (3.11 recommended, the Docker image uses python:3.11-slim), Node.js 20+ and npm, Docker (optional, for containerized deployment), 8GB+ RAM (16GB+ recommended), Ollama (required for both deployment methods).

There are three deployment methods:

1. Docker deployment: If you do not want to install Ollama on the host machine, you can use ./start-docker.sh (no arguments) to use local Ollama; if nothing is listening on port 11434, it will prompt you to switch to containerized Ollama, and adding -y (or setting NONINTERACTIVE=1) allows the script and CI to adopt this fallback without prompting.
2. Direct development (recommended for development): The run_system.py launcher manages four services and writes their PIDs to logs/run_system.pid—the Ollama server (port 11434, model serving, reused if already running), the RAG API server (port 8001, indexing, retrieval, and agent loop), the backend server (port 8000, sessions, indexing, uploads, chat history), and the frontend server (port 3000, Next.js Web interface, optional, skipped when npm is missing). At startup it checks whether qwen3.5:9b and qwen3.5:4b exist, and if missingit runs ollama pull.
3. Manual component-by-component startup: All commands must be run from the repository root, because relative paths (backend/chat_data.db, lancedb/, index_store/, shared_uploads/) are resolved based on the current working directory.

Installation notes: only two Ollama models need to be pulled explicitly; the embedding model (microsoft/harrier-oss-v1-0.6b, 1.2 GB) is downloaded from HuggingFace on first use; the reranker (about 7.5 GB) is lazily loaded and downloaded on the first query that requires reranking. All settings have usable defaults, so LocalGPT can run even without a .env; to override, create a .env in the repository root. NEXT_PUBLIC_* values are baked into the frontend bundle at next build time, so they require a rebuild after modification. Changing EMBEDDING_MODEL invalidates existing indexes, so the index must be rebuilt after switching embedding models.

Quick start flow: open http://localhost:3000，点击 "Create New Index", upload documents (PDF, DOCX, TXT, MD, HTML), configure processing options, and click "Build Index"; then create a chat session, select an index, ask questions in natural language, and get answers with source citations. An index's embedding model is a property of the index itself and must be selected when building the index.

Applicable Scenarios

Suitable for users who need localized, privacy-first document question answering and retrieval, for example, working with corpora such as technical RFCs, M&A documents, service manuals, HR policies, and project documentation. The repository also provides corresponding evaluation corpora.

Limitations

• The RAG API is single-threaded, and requests are serialized: only one chat or indexing task can run at a time (the backend gateway is multi-threaded).
• Vision/multimodal models are not within the pipeline; PDF parsing and OCR are handled by Docling. Models such as GLM-OCR or Qwen3-VL can be added as preprocessing steps, but are not currently integrated.
• If the reranker fails to load, the pipeline logs a warning and continues without reranking, rather than falling back to another model.
• Multi-vector (late-interaction) retrieval is experimental and controlled by an environment variable switch; it must be provided through an out-of-process sidecar; all have been tested in practice but none are enabled by default.
• Some components (Late Chunking, answer verification, cross-reference jumping, document upgrading, multi-vector retrieval) are disabled by default because they did not show sufficient benefits in evaluation.
• Embedding dimensions are not hardcoded, but measured from the vectors produced by the loaded model; appending vectors of different widths to an existing LanceDB table will cause an error and require rebuilding.

The above information comes from the project description; other details (such as specific pricing, billing models, etc.) are not yet provided.