h2ogpt

h2oGPT is an Apache V2 open-source project developed by H2O.ai, positioned for querying and summarizing documents using local private GPT large language models, or for directly chatting with a local private LLM.

Purpose It can be used to converse with local private GPT large language models, and can also query and summarize documents, covering many document types such as PDF, Excel, Word, images, video frames, YouTube, audio, code, text, and MarkDown. It is officially described as 100% private and Apache 2.0 licensed.

Main capabilities

• Private offline document database, supporting PDF, Excel, Word, images, video frames, YouTube, audio, code, text, MarkDown, etc.
• Persistent database using accurate embeddings (instructor-large, all-MiniLM-L6-v2, etc.) (Chroma, Weaviate, or in-memory FAISS).
• Efficient use of context with instruct-tuned LLMs (without LangChain's few-shot approach).
• Parallel summarization and extraction, with output reaching up to 80 tokens per second under the 13B LLaMa2 model.
• HYDE (Hypothetical Document Embeddings) enhanced retrieval based on LLM responses.
• Semantic Chunking to better split documents (requires GPU).
• Supports multiple models: LLaMa2, Mistral, Falcon, Vicuna, WizardLM, with support for AutoGPTQ, 4-bit/8-bit, LORA, etc.; GPU supports HF and LLaMa.cpp GGML models, CPU supports HF, LLaMa.cpp, and GPT4ALL models.
• Attention Sinks supports generation of arbitrary length (LLaMa-2, Mistral, MPT, Pythia, Falcon, etc.).
• Gradio UI or CLI, with streaming output for all models.
• Upload and view documents through the UI, with control over multiple collaborative or personal collections.
• Vision models: LLaVa, Claude-3, Gemini-Pro-Vision, GPT-4-Vision.
• Image generation: Stable Diffusion (sdxl-turbo, sdxl, SD3), PlaygroundAI (playv2), Flux.
• Speech STT: uses Whisper, supports streaming audio transcription.
• Speech TTS: uses MIT-licensed Microsoft Speech T5, multi-voice and supports streaming audio conversion; also supports MPL2-licensed TTS, including voice cloning and streaming audio conversion.
• AI assistant voice control mode, enabling hands-free control of h2oGPT chat.
• Bake-off UI mode, enabling simultaneous comparison of multiple models.
• Easily download model artifacts and control models such as LLaMa.cpp through the UI.
• Authentication in the UI via username/password, supporting native or Google OAuth; and retains state by user/password.
• Open Web UI can use h2oGPT as a backend, via the OpenAI Proxy.
• Supports chat completion streaming output via the OpenAI Proxy, document question answering using h2oGPT ingestion and DocTR advanced OCR, vision models, audio transcription (STT), audio generation (TTS), image generation, authentication, and state retention.
• Supports Linux, Docker, macOS, and Windows.
• Inference server support: oLLaMa, HF TGI server, vLLM, Gradio, ExLLaMa, Replicate, Together.ai, OpenAI, Azure OpenAI, Anthropic, MistralAI, Google, Groq.
• OpenAI-compatible server proxy API (h2oGPT can serve as a direct replacement for an OpenAI server): chat and text completion (streaming and non-streaming), audio transcription (STT), audio generation (TTS), image generation, embeddings, function tool calling with automatic tool selection, AutoGen code execution agent.
• JSON mode: strict schema control for vLLM via outlines; strict schema control for OpenAI, Anthropic, Google Gemini, and MistralAI models; for some older OpenAI or Gemini models, provides schema-controlled JSON mode when the model is sufficiently intelligent (such as gemini 1.5 flash); arbitrary models can be extracted via code blocks.
• Web search integrated with chat and document question answering.
• Agents: search, document question answering, Python code, CSV frames; provides high-quality agents through an OpenAI proxy server on a separate port; code-first agents can generate charts, conduct research, evaluate images via vision models, etc. (client code openai_server/openai_client.py), this item has no UI, API only.
• Uses reward models to evaluate performance; over 1000 unit and integration tests ensure quality, with tests taking over 24 GPU hours.

Usage It is officially recommended to use Docker on Linux, Windows, and MAC for full capabilities. The Linux Script also has full capabilities, while the Windows and MAC scripts have fewer capabilities than using Docker. Provides Docker build and run documentation (Linux, Windows, MAC), Linux installation and run documentation, Windows 10/11 installation scripts, MAC installation and run documentation, and a quick start for any platform. Startup documentation covers: chat completion streaming output, document question answering using h2oGPT ingestion and DocTR advanced OCR, vision models, audio transcription (STT), audio generation (TTS), image generation, authentication, and state retention. Run details include: Linux (CPU or CUDA), macOS (CPU or M1/M2), Windows 10/11 (CPU or CUDA), GPU (CUDA, AutoGPTQ, exllama) run details, CPU run details, CLI chat, Gradio UI, client API (Gradio, OpenAI-compatible), inference servers (oLLaMa, HF TGI server, vLLM, Groq, Anthropic, Google, Mistral, Gradio, ExLLaMa, Replicate, OpenAI, Azure OpenAI), building Python Wheel, offline installation, low memory, Docker. It also provides LangChain documentation support, comparisons with PrivateGPT, etc., file types supported by LangChain, CLI, database controldocumentation on system, FAQ, model usage instructions, adding LLM models (including using GGUF and Attention Sinks), adding embedding models, adding prompts, in-context learning, multi-GPU, low memory usage, environment variables, HTTPS access for server and client, useful links, fine-tuning, Triton, commercial viability, etc. Example test commands: pip install requirements-parser pytest-instafail pytest-random-order playsound==1.3.0; conda install -c conda-forge gst-python -y; sudo apt-get install gstreamer-1.0; pip install pygame; GPT_H2O_AI=0 CONCURRENCY_COUNT=1 pytest --instafail -s -v tests; test the openai server against an already running local server: pytest -s -v -n 4 openai_server/test_openai_server.py::test_openai_client; or adjust/run tests/test4gpus.sh to run tests in parallel.

Applicable scenarios Suitable for scenarios requiring private querying and summarization of local documents, chatting with a local private LLM, question answering on uploaded multi-type materials such as images/video frames/audio/YouTube/code, visual understanding, image generation, speech transcription and synthesis, web search, multi-model comparison and evaluation, and integration into existing applications via an OpenAI-compatible API.

Limitations and notes The model may have been trained on internet text and may contain bias, racism, offensive, or other inappropriate content; generated content may sometimes exhibit bias or produce offensive or inappropriate content. The model is an AI tool, not a human, and may produce erroneous, meaningless, or irrelevant answers; users should critically evaluate the generated content and decide on its use themselves. Users bear full responsibility for any consequences arising from the use of this tool. Users should use it responsibly and ethically, and not to promote hate speech, discrimination, harassment, or any illegal or harmful activities. If you encounter bias, offensive, or other inappropriate content, please report it to the repository maintainers through the provided channels. The developers reserve the right to modify or update this disclaimer at any time withoutprior notice. Regarding specific charges, prices, platform availability, performance metrics, etc., the official text does not provide them one by one, and they are not provided for now.