SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

HPC-Ops - Tencent Hunyuan's open-source industrial-grade high-performance large model inference operator library

Author: AI小集 · Original source: AI工具集

What is HPC-Ops

HPC-Ops is an industrial-grade high-performance large model inference operator library open-sourced by the Tencent Hunyuan AI Infra team, covering core modules such as Attention, MoE, GEMM, sampling, and communication fusion, with native support for BF16/FP8 and mixed precision. HPC-Ops is deeply optimized for GPUs such as NVIDIA H20. Dynamic scheduling and operator fusion increase end-to-end QPM by 30%+, and multiple performance metrics significantly surpass mainstream baselines such as vLLM, FlashInfer, and SGLang. It has already supported Tencent Hunyuan's large-scale production-grade inference services.

HPC-Ops

Main features of HPC-Ops

  • Dynamic Attention schedulingFor online inference workloads with mixed long and short requests, runtime tile-level dynamic task scheduling is used to achieve CTA-level load balancing, delivering up to 2.95x acceleration in long-text scenarios.
  • Router GEMMFP32-precision computation is simulated by combining dual BF16 GEMMs. Weights are split offline, and the inference stage is fused into a single kernel, balancing high precision for numerically sensitive scenarios with the high throughput of Tensor Cores.
  • FusedMoEThe five stages of routing indexing, Gate-Up GEMM, activation quantization, Down GEMM, and Top-K weighted aggregation are restructured into a bubble-free pipeline, eliminating memory movement and kernel launch overhead.
  • Fused AllReduce+NormCross-GPU communication, residual addition, and RMSNorm normalization are deeply fused, using NVLink multicast and P2P technology to achieve integrated communication and computation, alleviating tensor parallelism bottlenecks.
  • Sampler large operator fusionMore than 10 kernels, including repetition penalty, temperature scaling, Top-K, Top-P, Softmax, and random sampling, are fused into 2 CUDA kernels, addressing the shortcomings of post-processing at the end of inference.
  • GroupGEMM FP8Supports expert parallelism and grouped expert matrix multiplication, with native support for per-tensor and block-wise FP8 quantization, adapting to efficient inference for MoE models.

Technical principles of HPC-Ops

  • Attention dynamic schedulingTraditional static split-kv cannot handle the dynamic load of mixed long and short requests. HPC-Ops splits all requests by a unified tile granularity, evenly distributes the task size of each CTA through the global total number of tiles, and then uses a greedy bin-packing algorithm to achieve extreme equal distribution. The Task Assign module generates a dedicated task mapping table before each inference, and the Attention Kernel at each layer accurately claims tasks according to the table. Finally, the Combine Kernel merges the results, achieving load balancing across the entire process.
  • Router GEMM dual BF16 simulating FP32For precision-sensitive scenarios such as MoE routing, FP32 weights are split offline into high-order BF16 and low-order residual BF16 (scaling factor 1/256). During inference, two BF16 Tensor Core GEMMs are executed and linearly combined, while activations remain BF16 throughout. The dual-path computation is fused into a single kernel, inputs are moved only once, intermediate results are cached in dual register accumulators, and in the Epilogue stage a single FMA correction is applied before writing out, with no HBM round-trip overhead throughout.
  • FusedMoE pipeline restructuringIn the routing stage, shared-memory tiled statistics are used to reserve a contiguous memory output range for each expert; Gate-Up GEMM directly reads the original input through the routing index, eliminating independent Gather movement; Warp Specialization is removed, and the same Warp Group performs both data movement and computation, upgrading the memory latency hiding logic from an intra-CTA software pipeline to cross-CTA hardware scheduling; PDL technology connects the entire chain into bubble-free execution.
  • Fused AllReduce+Norm communication-computation fusionBased on CUDA multimem and P2P technology, encapsulated as an NVLink-native integrated operation RMSNorm(AllReduce(x) + residual, weight); the high-throughput version relies on the NVSwitch multicast mechanism to adapt to Prefill scenarios, while the low-latency version is based on the Lamport P2P mechanism and uses PDL to achieve dual-kernel overlapped execution, adapting to Decode scenarios.

How to use HPC-Ops

  • Hardware environment: Prepare a server equipped with an NVIDIA SM90 architecture GPU (such as H20), and install Python 3.8+, CUDA 12.8+, and a C++17 compiler.
  • Clone the repository: Clone the HPC-Ops source code repository from GitHub to your local machine.
  • Compile and package: Enter the project directory and run the compile and package command to generate the wheel installation file.
  • Install and deploy: Use the pip command to install the generated wheel file and complete the environment deployment.
  • Import the module: Import the hpc module in a Python script.
  • Prepare data: Prepare input tensors and configuration parameters that meet the precision requirements according to the target operator type.
  • Call the operator: Call the Python API of the corresponding operator (such as Attention, FusedMoE, Sampler, etc.) to perform high-performance inference.
  • Test and verify: Refer to the test scripts in the tests directory to verify operator correctness and specific usage.
  • Framework integration: Seamlessly embed HPC-Ops into mainstream inference frameworks such as vLLM and SGLang through the provided Python API.
  • Performance benchmark: Use the scripts in the benchmark directory to run performance reproduction and comparison tests on the target hardware.

Core advantages of HPC-Ops

  • End-to-end full-link optimization: It covers the full inference link of Attention → Router → MoE → communication fusion → Sampler, improving end-to-end QPM by 30%+, rather than optimizing a single operator.
  • Dynamic load balancing to fundamentally solve long-tail issues: Runtime tile-level dynamic scheduling and a greedy bin-packing algorithm accelerate long text by up to 2.95x in mixed long-and-short request scenarios, improving end-to-end QPM by 17%.
  • High-precision Router GEMM: Dual BF16 combination simulates FP32 precision, with an error only 1/36 of TF32, and performance up to 3.22x faster than cuBLAS FP32, balancing numerical sensitivity and compute release.
  • MoE bubble-free pipeline: FusedMoE reconstructs the five stages of routing, Gate-Up, quantization, Down, and Top-K aggregation into a single execution pipeline, eliminating memory movement and kernel launch overhead, achieving a 1.2x~1.6x improvement over vLLM/SGLang.
  • Deep integration of communication and computation: Fused AllReduce+Norm is encapsulated as an NVLink-native integrated operation, breaking through the tensor parallelism bottleneck and achieving up to 1.68x speedup over NCCL/FlashInfer.

HPC-Ops project address

  • GitHub repository：https://github.com/Tencent/hpc-ops

Comparison of HPC-Ops with similar competing products

Dimension HPC-Ops FlashInfer
Project nature An industrial-grade operator library open-sourced and long-term maintained by Tencent Hunyuan, verified through large-scale production A community-driven open-source high-performance operator library, widely integrated by frameworks such as vLLM and SGLang
Attention dynamic scheduling Supports runtime tile-level dynamic task allocation and greedy bin packing, accelerating long text by up to 2.95xin mixed long-and-short request scenarios, mixed batch acceleration 1.59x~1.76x It mainly adopts static split-k scheduling, which performs stably under uniform-length loads, but is prone to CTA-level long-tail latency when long and short sequences are mixed
Standard Attention Performance BF16 Prefill up to 1.33x、Decode 2.22x；FP8 Decode 2.0x(relative to the FlashInfer baseline) As the community's mainstream baseline, it delivers excellent performance, but there is room for optimization under dynamic loads and sparse scenarios
Sparse Attention FP8 block-sparse Prefill, supporting precomputed block masks to skip irrelevant KV Tiles, up to 3.16x Supports block-sparse mode, but HPC-Ops has a more refined implementation of sparse scheduling and Tile quantization under FP8 precision
MoE Fusion FusedMoE reconstructs routing index → Gate-Up → quantization → Down → Top-K aggregation into a bubble-free pipeline, leading in performance 1.2x~1.6x Provides basic MoE support, without full-module pipeline fusion, and incurs multi-stage memory movement and Kernel launch overhead
Router GEMM Its original dual BF16 simulates FP32 precision, with an error only 1/36 of TF32, and performance 3.22x, suitable for numerically sensitive routing modules No dedicated high-precision Router GEMM optimization; it must rely on cuBLAS FP32 or accept BF16/TF32 precision loss
Communication-Computation Fusion Fused AllReduce+Residual+RMSNorm is encapsulated as a native NVLink integrated operation, up to 1.68x~1.76x Provides basic communication operators, but no native fusion implementation of AllReduce, residual, and normalization

Application Scenarios of HPC-Ops

  • High-Throughput Online Inference Services: Dynamic Attention scheduling and Sampler fusion significantly reduce long-tail latency, making it suitable for real business workloads with mixed long and short requests.
  • MoE Large Model Inference: FusedMoE is deeply optimized for MoE models such as DeepSeek-V3 and Qwen3-235B, improving inference efficiency in TP/EP parallel scenarios.
  • Tensor-Parallel Distributed Inference: Fused AllReduce+Norm breaks through the multi-card communication bottleneck, making it suitable for large model deployment with multiple GPUs on a single node.
  • Precision-Sensitive Inference: Router GEMM achieves FP32 precision with BF16 compute power, meeting the needs of numerically sensitive modules such as MoE routing and sparse Attention.