TensorRT-LLM is an open-source library released by NVIDIA for optimizing inference of large language models (LLM) and visual generation (Visual Gen). It provides advanced optimization capabilities for NVIDIA GPUs, helping users achieve better performance in inference scenarios.
Main capabilities
- Custom kernels for common inference operations, covering attention, GEMM, MoE, etc.
- Runtime algorithm optimizations, including Prefill-Decode disaggregation, Wide Expert Parallelism, Speculative Decoding, etc.
- High-level Python LLM API, supporting various inference setups from single GPU to multi-GPU, multi-node deployment.
- Built-in support for multiple parallel strategies and advanced features.
- Seamless integration with inference ecosystems such as NVIDIA Dynamo and Triton Inference Server.
- Modular and easy to modify: the PyTorch-native architecture allows developers to experiment with the runtime or extend functionality; multiple popular models are predefined and can be customized with native PyTorch code.
Usage Official documentation provides the following entry points:
- Quick Start Guide
- Running DeepSeek
- Installation Guide
- Supported Hardware, Models, and other Software
- Benchmarking Performance
TensorRT-LLM provides Python and C++ runtime components for orchestrating inference execution in a high-performance manner.
Applicable scenarios
- Deploying LLM inference on NVIDIA GPUs, from single GPU to multi-GPU, multi-node scenarios.
- Scenarios requiring advanced inference optimizations such as custom kernels, quantization, speculative decoding, long-context optimization, MoE optimization, etc.
- Used in conjunction with inference serving ecosystems such as Triton Inference Server and NVIDIA Dynamo.
- Visual generation (diffusion model) inference.
Latest updates (some updates listed officially)
- KV Cache Compression in TensorRT LLM
- GVR V2：Self-Sampling and Multi-Thresholding for Faster Exact Top-K
- Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention
- Evaluating Agentic Serving with Trace Replay and Job-Level Metrics
- DeepSeek-V4 on NVIDIA Blackwell：Model-Specific and Agentic-Workload Optimizations
- Scaling Video Generation Across NVL72 Rack
- Joint Optimization of Agent Applications and TensorRT-LLM
- Tuning CUDA Graph Batch Sizes for Higher Output Throughput
- DWDP：Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
- Optimizing MoE Communication with One-Sided AlltoAll Over NVLink
- Sparse Attention in TensorRT LLM
- Accelerating Long-Context Inference with Skip Softmax Attention
- Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs
- Inference Time Compute Implementation
- Combining Guided Decoding and Speculative Decoding
- ADP Balance Strategy
- Running a High-Performance GPT-OSS-120B Inference Server
- Scaling Expert Parallelism (Part 1 and Part 2)
- N-Gram Speculative Decoding
- Disaggregated Serving
- Support for diffusion models for visual generation
- Day-0 support for OpenAI GPT-OSS-120B and GPT-OSS-20B
- Day-0 support for LG AI Research EXAONE 4.0
- Meta Llama 4 Maverick exceeds 1,000 TPS/User on Blackwell
- Llama 4 exceeds 40,000 tokens/second on B200 GPU
- Fully open source, development moved to GitHub
- Official release of DeepSeek R1 performance benchmarking best practices
- NVIDIA Blackwell achieves world-record DeepSeek-R1 inference performance with TensorRT LLM
- Earlier updates also include KV Cache Reuse optimization, Bing search optimization, Llama 3.3 70B inference throughput improvement, Speculative Decoding support, Jetson AGX Orin support, Multiblock Attention, multilingual optimization, SD3 INT8 quantization, ComfyUI deployment, etc.
Deprecation policy Starting from version 1.0, TensorRT LLM adopts a deprecation policy: deprecation notices are recorded in the Release Notes; deprecated APIs, methods, classes, or parameters are marked in the source code with the deprecation time, and a runtime deprecation warning is generated when used. After deprecation, a 3-month migration period is provided, during which deprecated items remain usable but generate warnings; after the migration period ends, they are removed according to semantic versioning rules (major version changes may include breaking removals). The scope of deprecation includes overall API/method/class deprecation, as well as deprecation of only some parameters (the method itself remains usable, and the deprecated parameters will be removed in a future version).
Telemetry data collection TensorRT-LLM defaultCollects anonymous telemetry data to understand usage patterns in aggregate and determine engineering priorities; the data cannot be traced back to any individual user. It does not collect prompts, outputs, model weights, model paths, tokenizer paths, personally identifiable information, raw free-form configuration strings, or persistent identifiers; deployment identifiers are temporary, randomly generated per deployment, and not associated with users. The collected data includes: entry point (such as LLM API, CLI, serve command), deployment duration (via periodic heartbeats), GPU SKU, count, memory and CUDA version, public model architecture identifiers (which may be upstream checkpoint architectures or publicly documented TensorRT-LLM runtime architectures; identifiers outside the allowlist are sent only as deterministic SHA-256 hashes), parallelism configuration (TP/PP/CP/MoE-EP/MoE-TP sizes), quantization algorithm, dtype, KV cache dtype, system information (OS platform, Python version, CPU architecture, CPU count), TRT-LLM version and backend, feature summary flags (LoRA, speculative decoding, prefix caching, CUDA graphs, chunked context, data parallelism), disaggregated serving metadata (role and deployment ID), and process-local LLM lifecycle information.
Limitations and Notes The materials do not provide specific pricing, platform support details, a complete evaluation of performance numbers, or specific usage steps; related content is not provided for now.
