GuideLLM is an SLO-aware benchmarking and evaluation platform for real-world LLM inference needs, maintained by the vLLM project (repository address:https://github.com/neuralmagic/guidellm）。它的目标是在接近生产环境的条件下，评估语言模型在不同工作负载和配置下的表现，帮助团队理解系统行为、资源需求与运行极限。
Purpose
GuideLLM is used to evaluate the performance, efficiency, and reliability of LLM deployments under real workloads. It simulates end-to-end interactions with OpenAI-compatible servers and vLLM native servers, generates workload patterns that reflect production usage, and produces detailed reports for engineering and machine learning teams to perform model behavior evaluation, deployment tuning, and capacity planning.
Key Capabilities
- Captures complete latency and token-level statistics, supporting SLO-driven evaluation, including TTFT, ITL, and the full distribution of end-to-end behavior.
- Generates realistic, configurable traffic patterns, covering synchronous, concurrent, and rate-based modes, and supports reproducible sweeps for identifying safe operating ranges.
- Supports both real datasets and synthetic multimodal datasets, enabling controlled experiments and production-style evaluation within the same framework.
- Produces standardized, exportable reports for dashboards, analysis, and regression tracking, ensuring consistency across teams and workflows.
- Provides high-throughput, scalable benchmarking capabilities, supports multiprocessing, multithreading, and asynchronous execution, and offers a flexible CLI/API for customization or quick start.
Usage
Installation
Prerequisites: The operating system is Linux or MacOS, and the Python version is 3.10 to 3.13. The latest version of GuideLLM can be installed from PyPI via pip, or installed from source, or run from the latest container image at ghcr.io/vllm-project/guidellm. The published images are multi-architecture (linux/amd64 and linux/arm64) and are suitable for Docker, Podman, Kubernetes, and OpenShift; for production environments, it is recommended to pin release tags and not use architecture-specific tags such as vX.Y.Z-amd64 / vX.Y.Z-arm64 as image references.
Start the inference server
Start any OpenAI-compatible endpoint, such as vLLM, and confirm that the server is running at http://localhost:8000。
Run the first benchmark
Run a sweep to identify the model's maximum performance and maximum rate, during which progress updates and a summary of each benchmark will be displayed.
View the output
After the benchmark is complete, GuideLLM writes benchmarks.json and benchmarks.csv by default, with the files saved in the directory configured by GUIDELLM__DEFAULT_RESULTS_DIR, or in the current directory when this variable is not set, while also printing a summary and the location of the generated files to the console. Use --output to replace the default JSON and CSV output, and it supports generating formats such as HTML.
Output files and reports
JSON or YAML is used for detailed analysis, CSV for spreadsheet comparison, HTML for self-contained visual reports, and PLOT for static performance charts. The console provides a summary of each benchmark, and its table can be copied into spreadsheet software using | as the delimiter.
Common use cases and configurations
- Traffic patterns:
--profile kind=<type>defines the traffic pattern, with options synchronous, concurrent, throughput, constant, poisson, or sweep; constant/poisson uses rate to set requests per second, concurrent uses streams, and throughput uses max_concurrency. - Constraints:
--constraint kind=max_duration,seconds=<seconds>or--constraint kind=max_requests,count=<count>limits each strategy by time or number of requests;--constraint kind=max_errors,count=<count>stops when the total number of errors exceeds the threshold;--constraint kind=over_saturationenables oversaturation detection. - Dataset sources: Supports HuggingFace datasets, local files, and synthetic data, such as synthetic_text, huggingface, json_file, csv_file, text_file, trace_synthetic, etc., and multiple sources can be specified repeatedly;
--data-column-mapperis used for field mapping;--data-loader type=pytorch,samples=1000limits the number of rows loaded (-1 means all);--tokenizer huggingface_auto "model=gpt2"For synthetic data or local token counting. - Synthetic visual data:
synthetic_imageandsynthetic_videoThe two --data types can be combined with synthetic_text for benchmarking vision-language model (VLM) serving configurations. - Request type and API target:
--backendConfigure the backend type and connection settings, including the target OpenAI-compatible endpoint URL and request_format (such as /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/audio/transcriptions, etc.). - Scenario:
--config(aliases --scenario, -c) Use a built-in scenario name or a custom scenario file path; CLI options override scenario defaults. - Benchmark control: The profile configuration supports warmup and cooldown (percentage or absolute units) to exclude the effects of initialization and shutdown, ensuring measurement stability and repeatability.
Use cases
GuideLLM is suitable for teams that need to evaluate the performance, efficiency, and reliability of LLMs in production-like environments, including: benchmarking OpenAI-compatible or vLLM native services; determining safe operating ranges through reproducible sweeps; conducting controlled experiments using real or synthetic data (including multimodal inputs); generating standardized reports for dashboards, analysis, and regression tracking; and providing a basis for capacity planning.
Limitations
The materials do not provide specific performance data, pricing, or commercial fee information. Some specific configuration details and the complete list of parameters can be found through guidellm run --help and the official documentation. Development and contribution workflows can be found in DEVELOPING.md, CONTRIBUTING.md, and CODE_OF_CONDUCT.md. The complete documentation covers installation, backend configuration, dataset processing, metric definitions, output formats, tutorials, and an architecture overview. The project is licensed under the Apache License 2.0.
