FasterTransformer is a Transformer inference optimization project tested and maintained by NVIDIA, primarily providing highly optimized inference implementations for the encoder and decoder components in natural language processing. The project source code is based on C++, with underlying dependencies on CUDA, cuBLAS, and cuBLASLt, and can automatically leverage Tensor Cores on Volta, Turing, and Ampere GPUs to accelerate computation on FP16-precision data and weights.
In terms of use, FasterTransformer focuses on inference acceleration for Transformer models. The encoder and decoder are two important components in NLP, and the Transformer layer has become the mainstream architecture for both; FasterTransformer implements highly optimized Transformer layers for these two types of components.
Main capabilities include: providing at least one API among the TensorFlow, PyTorch, and Triton backends, allowing users to integrate FasterTransformer directly into these frameworks, with official example code demonstrating usage and performance; supporting models such as BERT and GPT, and covering multiple data types such as FP16, INT8, bfloat16, and FP8 (experimental); supporting multiple inference features such as multi-node multi-GPU inference, tensor parallelism, pipeline parallelism, streaming generation, beam search, sampling, MoE, soft prompts, and prefix prompts. All model support is implemented at the C++ level.
In terms of usage, users can integrate through the example code provided in the repository and the corresponding framework APIs. Detailed guides for each model are stored in the xxx_guide.md files under the docs/ directory (where xxx is the model name), and the documentation for the encoder and BERT is combined in bert_guide.md. Frequently asked questions are in docs/QAList.md. The repository also provides several environment variables for debugging and testing: FT_LOG_LEVEL controls the debug log level; when FT_NVTX is set to ON, nvtx markers can be inserted to assist performance analysis; when FT_DEBUG_LEVEL is set to DEBUG, cudaDeviceSynchronize() is executed after each kernel, making it easier to locate the point of error, but it significantly affects performance and is recommended only for debugging.
In terms of applicable scenarios, this project is suitable for developers who need efficient inference of Transformer models on NVIDIA GPUs, covering tasks such as machine translation and text generation. Performance data is tested based on hardware configurations such as 8xA100-80GBs and T4.
In terms of limitations, the official documentation clearly states that FasterTransformer development has migrated to TensorRT-LLM, and the original repository will no longer have subsequent development; developers are encouraged to use TensorRT-LLM to obtain the latest improvements in LLM inference. In addition, some features have hardware limitations; for example, the custom all reduce kernel only supports a tensor parallelism size of 8 on DGX-A100, and only supports CUDA using cudaMallocAsync. For other more specific limitations, it is recommended to refer to the official documentation.
