Purpose: TransformerEngine (TE) is a library released by NVIDIA for accelerating the training of Transformer models on NVIDIA GPUs, while also optimizing performance and memory usage for inference scenarios.
Main capabilities: It provides optimized basic modules and fused kernels, and works with automatic mixed-precision-style PyTorch and JAX APIs, so that low-precision training can be adopted without rewriting the training stack. The library manages the scaling factors, amax history, and quantization metadata required for low-precision recipes. Modules cover attention, linear layers, normalization, Mixture-of-Experts (MoE), and communication operations in large-scale distributed training. Precision support includes: FP8 training on Hopper, Ada, Blackwell, and Rubin GPUs; MXFP8 and NVFP4 training on Blackwell GPUs; FP16 and BF16 optimization on Ampere and later architectures; and support for tensor, sequence, context, and EP parallelism and communication overlap.
Usage: Install the latest stable version according to the framework you use; NVIDIA recommends using the NVIDIA NGC framework container to obtain an environment that can run directly, replacing <YY.MM> with the container release version listed in the NVIDIA Deep Learning Frameworks Support Matrix. Then refer to the PyTorch and JAX getting started guides; prerequisites, building from source, environment variables, and troubleshooting are covered in the installation guide. The project has been integrated with frameworks such as Hugging Face Accelerate, NVIDIA BioNeMo Recipes, NVIDIA JAX Toolbox, NVIDIA Megatron-LM, NVIDIA NeMo Megatron Bridge, and Amazon SageMaker Model Parallel Library.
Applicable scenarios: For teams conducting large-scale Transformer training and inference on NVIDIA GPUs, especially PyTorch or JAX users who want to adopt FP8/FP4 low precision to improve performance and reduce memory usage.
Limitations: Specific hardware and precision support is limited by GPU architecture (for example, FP8 requires Hopper, Ada, Blackwell, Rubin;MXFP8 and NVFP4 require Blackwell). The materials do not provide specific fee or pricing information.
