LLaMA Box is a language model inference server built on llama.cpp and stable-diffusion.cpp, provided purely in API form and containing no frontend assets. The project is no longer actively maintained, and the official recommendation is to consider using llama.cpp directly.
In terms of main capabilities, LLaMA Box is compatible with a variety of OpenAI-style API interfaces, including the Chat API, Chat Vision API, Chat Audio-In API (experimental), Function Calling API, Embeddings API, Images API, and the legacy Completions API, while also being compatible with the Jina Rerank API. In terms of vision model support, it covers the LLaVA series, MiniCPM VL series, Qwen2 VL series, GLM-Edge-V series, Granite VL series, Gemma3 VL series, SmolVLM series, Pixtral series, MobileVLM series, Mistral Small 3.1, Qwen2.5 VL series, InternVL2/InternVL3 series, LLaMA4 series, Qwen2.5-Omni series, and others. For audio input, it supports the UltraVox series, Qwen2 Audio series, Qwen2.5-Omni series, Voxtral series, and others. Function calling supports the LLaMA3 series, Granite series, Mistral series, Hermes2/Qwen2 series, ChatGLM4 series, DeepSeekV3 series, CommandR series, FunctionaryV3 series, DeepSeekR1 series, and others.
In terms of usage, users can download LLaMA Box from the latest release page. It supports platforms such as CentOS 7, Ubuntu 18.04/20.04/22.04, RockyLinux 8.9, macOS 13.7.5/14.7.5, and Windows Server 2022. Example startup command: llama-box -c 8192 -np 4 --host 0.0.0.0 -m <model path>. It can be called via curl at the http://localhost:8080/v1/chat/completions interface, or by using the chat.sh tool included with the project.
Other features include: support for speculative decoding (draft model or n-gram lookup); support for RPC server mode as a remote inference backend; support for splitting and offloading layers across multiple devices (including remote RPC servers), with both non-image models and image models able to be allocated via the --tensor-split parameter; support for injecting the X-Request-ID HTTP header for request tracing; and support for the X-Request-Tokens-Per-Second HTTP header to limit tokens per second.
Applicable scenarios include local or remote deployment of large language model inference services, multimodal (vision/audio) inference, function calling, image generation and editing (such as stable-diffusion-3.5-medium, FLUX.1-Fill-dev), and others. In terms of limitations, the project has stopped active maintenance, some features are marked as experimental, and specific performance and results are not yet provided.
