What is MT-TransformerEngine
MT-TransformerEngine is Moore Threads' open-source efficient training and inference optimization framework, designed specifically for Transformer models. The framework fully leverages the computing potential of Moore Threads' full-featured GPUs through techniques such as operator fusion and parallel acceleration, significantly improving training efficiency. It supports FP8 mixed-precision training, further optimizing performance and stability with the help of the GPU's native FP8 computing capability. MT-TransformerEngine works with MT-MegatronLM to achieve efficient hybrid parallel training, suitable for large models such as BERT and GPT.
Main features of MT-TransformerEngine
- Efficient training acceleration: by fusing multiple compute-intensive operations, it reduces memory access and computational overhead, significantly improving training efficiency. : it supports data parallelism, model parallelism, and pipeline parallelism, fully utilizing the computing resources of GPU clusters.
- Inference optimization: it optimizes the inference stage of Transformer models, reducing latency and improving throughput. By optimizing memory management, it reduces memory usage during inference.
- Collaboration with ecosystem tools
- Collaboration with MT-MegatronLM: it supports working with the MT-MegatronLM framework to achieve more efficient hybrid parallel training.
- Integration with MT-DualPipe: combined with MT-DualPipe, it further optimizes the utilization of computing resources.
- Support for Torch-MUSA: relying on the Torch-MUSA deep learning framework and the MUSA software stack, it is compatible with a variety of algorithms.
- Multimodal model support: it supports training multimodal Transformer models, suitable for complex tasks involving multiple modalities such as text and images.
- Communication optimization: by optimizing communication strategies, it reduces communication latency between GPUs and improves overall training efficiency.
Technical principles of MT-TransformerEngine
- Operator fusion: MT-TransformerEngine performs various operator fusion optimizations tailored to the characteristics of Transformer models. It fuses normalization layers with QKV horizontally, fuses self-attention computation, and fuses residual connections with fully connected layers, reducing the number of memory accesses and the overhead of Compute Unified Architecture (CUDA) kernel launches during computation, improving model inference performance.
- Parallel acceleration strategies
- Hybrid parallel training: it supports multiple parallel strategies such as data parallelism, tensor parallelism, and pipeline parallelism. Tensor parallelism achieves distributed computing across multiple cards by partitioning the matrix computation dimensions of attention layers and multilayer perceptrons; pipeline parallelism divides the model into multiple stages and improves throughput through microbatch passing.
- Reducing bubble rate: by integrating with technologies such as MT-DualPipe and DeepEP, it significantly reduces 'pipeline bubbles' and further improves parallel training efficiency.
- FP8 mixed-precision training: it deeply integrates the FP8 mixed-precision training strategy natively supported by Moore Threads GPUs. During training, it accelerates training based on low-precision FP8 computation while maintaining numerical stability through specific technical means, reducing memory usage, and significantly improving training efficiency.
- High-performance operator library: it integrates the high-performance operator library muDNN, deeply optimized for GPUs to improve computational efficiency.
Project address of MT-TransformerEngine
- Github repository：https://github.com/MooreThreads/MT-TransformerEngine
Application scenarios of MT-TransformerEngine
- Large-scale language model training: MT-TransformerEngine is suitable for training large-scale language models such as GPT, BERT, T5, etc. Through its efficient parallelization technology and FP8 mixed-precision training strategy, it can efficiently train models with billions or even hundreds of billions of parameters on large-scale GPU clusters.
- Multimodal model training: The framework supports the training of multimodal Transformer models and can handle data containing multiple modalities such as text, images, and video. For example, Meta-Transformer uses an Encoder with frozen weights to perform multimodal perception without paired multimodal training data.
- Real-time inference: In real-time inference scenarios that require low latency, MT-TransformerEngine can significantly improve inference speed through an optimized inference engine and FP8 precision, making it suitable for tasks such as natural language processing and image recognition.