exllama

ExLlama is an inference implementation for Llama models with 4-bit GPTQ quantized weights, written in Python, C++, and CUDA, as a more memory-efficient rewrite of the Llama implementation in Hugging Face Transformers. The project aims to provide fast and memory-efficient inference on modern GPUs.

Purpose

ExLlama is mainly used to run GPTQ 4-bit quantized Llama series models, suitable for scenarios that require efficient inference of large language models on consumer-grade GPUs. By optimizing memory usage and computation speed, it makes it possible to run larger models under limited VRAM.

Main capabilities

• Supports 4-bit GPTQ quantized weights, significantly reducing VRAM usage.
• Optimized for NVIDIA 30 series and newer GPUs (such as RTX 4090, RTX 3090-Ti), providing high-performance inference.
• Provides a standalone Python module for easy integration into other projects.
• Includes a simple Web UI, supports multi-bot mode, and sessions are stored by default in the ~/exllama_sessions/ directory.
• Supports sharded models and can handle multiple .safetensors files.
• Provides Docker support, allowing the Web UI to run in an isolated container (NVIDIA GPU only), with rootless mode support.
• ExLlamaV2 has been initially released, with improved features.

Usage

Environment requirements

• Python 3.9 or later.
• PyTorch 2.0.1 or 2.1.0 (nightly), with cu118.
• safetensors 0.3.2.
• Only the Web UI requires additional dependencies (Linux/WSL or Windows prerequisites).

Windows installation (non-WSL)

1. Install MSVC 2022 (Visual Studio 2022 IDE or Build Tools for Visual Studio 2022, make sure to check "Desktop development with C++").
2. Install a suitable version of PyTorch, choosing the CUDA version (stable version 2.0.1 is also fine).
3. Install CUDA Toolkit (either 11.7 or 11.8 is fine, and it must match the Compute Platform version of PyTorch).
4. For best performance, enable hardware-accelerated GPU scheduling.

Clone and run

• Clone the repository, install dependencies, and run benchmarks.
• CUDA extensions are loaded at runtime and do not need to be installed separately. On the first run, they will be compiled and cached to ~/.cache/torch_extensions/, which may take some time.
• A chatbot example is provided.

Web UI

• It can be used through a simple Web UI and supports multi-bot mode.
• Sessions are stored by default in ~/exllama_sessions/, and the location can be changed with -sd.
• It can be run in isolation through a Docker container, requires the NVIDIA Container Toolkit, and rootless mode is recommended.
• Use docker compose to build the image: set the MODEL_PATH and SESSIONS_PATH variables in the .env file, then run the docker compose command. It can also be built manually.
• By default, the service in the container runs as a non-root user and will change the ownership of the bind-mounted directory; this can be disabled by setting RUN_UID=0.
• The Web UI can be accessed through http://localhost:5000 Access.
• The configuration can be viewed in docker-compose.yml and modified via docker-compose.override.yml.

Use Cases

• Run 4-bit quantized Llama models for inference and conversation on a local GPU.
• Scenarios requiring low VRAM usage and high inference speed.
• Integrate into other Python projects.
• Interactive conversation through a Web UI, supporting multiple bots.
• Deploy in a Docker environment to enhance security and ease of deployment.

Limitations

• The project is still in progress and may be incomplete.
• Only supports 30-series and newer NVIDIA GPUs; performance on Pascal or older GPUs is poor, and AutoGPTQ or GPTQ-for-LLaMa is recommended.
• ROCm (AMD GPU) is theoretically supported (via HIP), but the author has no AMD device for testing or optimization.
• Cannot reach the full sequence length without OOM.
• VRAM usage is affected by PyTorch internal overhead and is difficult to predict precisely; it is recommended to optimize model VRAM usage to around 20 GB on a 24 GB GPU.
• Perplexity measurement is only used to verify that the model works; the dataset is a small sample of WikiText, and the scores cannot be directly compared with other Llama benchmarks.
• The Docker image currently only supports NVIDIA GPUs.
• Model support list and changelog: 2023-01-09 added rope_theta parameter support for CodeLlama (partial), 2023-08-09 added support for sharded models, 2023-08-12 initial release of ExLlamaV2.
• The official list of confirmed usable models is provided in the repository.

Category

Based on its purpose, ExLlama belongs to AI development platforms (ai-frameworks), because it is a framework/implementation for model inference.