vLLM is a fast and easy-to-use LLM inference and serving library, positioned as a simple, fast, and low-cost LLM serving solution for everyone. The project was initially developed at the Sky Computing Lab at UC Berkeley and has now grown into one of the most active open-source AI projects, maintained by more than 2,000 contributors, numerous academic institutions, and enterprises.
In terms of use, vLLM is mainly used for high-throughput inference and online serving deployment of large language models, and is suitable for scenarios that require efficient utilization of GPU/CPU resources and support for a large number of concurrent requests.
Main capabilities include:
- High-throughput serving: industry-leading serving throughput.
- Memory management: efficient management of attention key-value memory through PagedAttention.
- Batching and caching: support for continuous batching, chunked prefill, and prefix caching.
- Model execution: fast and flexible execution using piecewise and full CUDA/HIP graphs.
- Quantization: support for FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and more.
- Optimized kernels: including attention kernels such as FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton, as well as GEMM/MoE kernels using CUTLASS, TRTLLM-GEN, and CuTeDSL.
- Speculative decoding: support for n-gram, suffix, EAGLE, and DFlash.
- Compilation optimization: automatic kernel generation and graph-level transformations using torch.compile.
- Disaggregated architecture: support for disaggregated deployment of prefill, decode, and encode.
In terms of usage, vLLM integrates seamlessly with Hugging Face models, supports multiple decoding algorithms such as parallel sampling and beam search, supports tensor, pipeline, data, expert, and context parallelism for distributed inference, supports streaming output, can generate structured output using xgrammar or guidance, supports tool calling and reasoning parsers, provides an OpenAI-compatible API server as well as Anthropic Messages API and gRPC support, and supports efficient multi-LoRA for dense and MoE layers. Installation can be done via uv (recommended) or pip , or it can be built from source for development.
Applicable scenarios include: production environments that require high-throughput LLM serving, multimodal model inference, embedding and retrieval model deployment, running reward and classification models, and distributed inference, among others.
In terms of hardware support, vLLM supports NVIDIA GPU, AMD GPU, Intel GPU, and x86/ARM/PowerPC CPU, and can support Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more through hardware plugins.
In terms of model support, vLLM seamlessly supports more than 200 model architectures on Hugging Face, including decoder-only LLMs (such as Llama, Qwen, Gemma), mixture-of-experts LLMs (such as Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS), hybrid attention and state space models (such as Mamba, Qwen3.5), multimodal models (such as LLaVA, Qwen-VL, Pixtral), embedding and retrieval models (such as E5-Mistral, GTE, ColBERT), and reward and classification models (such as Qwen-Math).
In terms of limitations, the materials do not provide specific performance upper limits, minimum hardware configuration requirements, or pricing information, and none is currently provided.
