ZhiLight is a highly optimized large language model (LLM) inference acceleration engine jointly developed by Zhihu and ModelBest Inc. The "Zhi" in its name stands for Zhihu. The engine is designed to accelerate inference for models such as Llama and its variants, and it has significant performance advantages especially on PCIe-based GPUs. According to official statements, compared with mainstream open-source inference engines such as vLLM, ZhiLight has clear performance advantages.
In terms of main capabilities, ZhiLight provides a rich set of features: it adopts an asynchronous OpenAI-compatible interface adapted from vLLM; supports custom tensors and unified global memory management; implements an encoding and all-reduce overlap mechanism called "dual streams," and supports Int8 quantized all-reduce to further reduce all-reduce overhead; supports host-side all-reduce based on SIMD instructions; provides optimized fused kernels, such as qkv, residual, and layernorm; supports decoding fused batch attention based on tensor core instructions; supports tensor parallelism (TP) and pipeline parallelism (PP) on a single node, with TP recommended; supports dynamic batching, flash attention prefill, chunked prefill, and prefix cache; supports Native INT8, SmoothQuant, FP8, AWQ, and GPTQ quantization; supports GPTQ's Marlin kernel; supports MoE, DeepseekV2 MoE, and DeepseekV2 MLA; supports Llama/Llama2, Mixtral, Qwen2 series, and similar models.
In terms of version updates, in May 2025 it supported DeepSeek-V3/DeepSeek-R1 FP8 block-quantized models; in February 2025 it supported DeepSeek-V3/DeepSeek-R1 AWQ/GPTQ quantized models; in January 2025 it supported the DeepSeek-VL2 multimodal models (deepseek-vl2-tiny, deepseek-vl2-small, deepseek-vl2), and supported LLaMA3 models (Llama-3.1-8B, Llama-3.1-70B, Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B).
In terms of usage, ZhiLight depends only on the CUDA runtime, cuBLAS, NCCL, and a small number of Python packages in requirements.txt. It can be run or built through the official Docker image, or you can directly refer to docker/Dockerfile.
In terms of performance explanation, the official team conducted performance evaluations on a variety of mainstream NVIDIA GPUs for different model sizes and precisions. On PCIe devices, for dense models with parameter sizes ranging from 2B to 110B, ZhiLight shows significant performance advantages compared with mainstream open-source inference engines. The test metrics include: QPS (queries per second), TTFT (time to first token), and TPOT (time per output token). The test environments include AD102 PCIe (a consumer-grade GPU used for experimental research), A800 (a data center GPU used for production deployment), and H20 (a data center GPU used for production deployment). The tested models include large-scale models: DeepSeek-R1, Qwen1.5-110B, Qwen2-72B, LLama-3.1-70B; medium-scale models: Qwen2.5-14B, Llama-3.1-8B, Minicpm-2B. The compared inference engines include vLLM, SGLang, and ZhiLight. More benchmark tests can be viewed in benchmarks.
The project's license is the Apache License 2.0. Information such as specific installation commands, configuration methods, detailed performance numbers, charging models, and platform restrictions is not provided in the materials.
