SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillskb-accel-sdk

kb-accel-sdk

Agents & models

View repository

Updated 10-15 20:08

About this skill

Accelerator Platform SDK and Ecosystem Knowledge Base

Knowledge-type skill: content is cross-platform public knowledge (research date 2026-09-27; Ascend section refreshed 2026-09-28 with verification via 910B2 real-machine initialization; entries prone to becoming outdated include re-check entry points). Machine-specific facts are not written here (managed at the host layer).When conclusions conflict with the real machine, the real machine prevails, and updates are fed back into this library.

Platform Overview

Platform Vendor Software Stack smi torch Backend Communication Library wheel Source
CUDA NVIDIA Driver + CUDA Toolkit (nvcc/cuBLAS/cuDNN/NCCL/TensorRT) nvidia-smi Native NCCL PyPI / download.pytorch.org/whl/cuXXX
MUSA Moore Threads mtgpu driver + MUSA SDK (musart/toolkit/muDNN/MCCL) mthreads-gmi torch_musa MCCL dl.mthreads.com vendor index
Ascend Huawei Driver + firmware + CANN (toolkit/kernels/NNAL, including HCCL, ATB; inference additionally has MindIE) npu-smi torch_npu HCCL PyPI(torch_npu) + Huawei Cloud mirrors.huaweicloud.com/ascend (triton-ascend/offline packages)
ROCm AMD amdgpu driver + ROCm (HIP/hipBLAS/MIOpen/RCCL) rocm-smi Native (HIP translation, device is still called cuda) RCCL download.pytorch.org/whl/rocmX.Y
PPU Alibaba T-Head Zhenwu 810/810E, GPGPU route; very few public SDK documents (existence only officially confirmed in 2026-01) (unverified) (unverified) (unverified) Alibaba Cloud channel, no public source
MetaX MACA MetaX MXMACA（mcBLAS/mcDNN/mcFlashAttention） mx-smi mcPytorch mcCL (unverified) MetaX developer community mirror/release packages

Cambricon MLU (Neuware/torch_mlu) and Hygon DCU (DTK, ROCm-based) are also frequently encountered; keep a quick note for reference.

Ecosystem Matrix (Key Pitfall-Prevention Table)

Framework CUDA ROCm Ascend MUSA PPU
triton Upstream Native Upstream Native triton-ascend（fork；Mutually exclusive with upstream triton under the same name, see Ascend section) Triton-MUSA(fork, now at 3.6; on PyPI with the same name as upstream, installing the wrong one does not error) unverified
transformers Native Native Upstream support (torch_npu) Upstream declares support (torch_musa README, but fused kernels/optional extensions are not guaranteed) unverified
tilelang Upstream Native Upstream (HIP) tilelang-ascend（tile-ai org） tilelang_musa(vendor index) unverified
vllm Upstream Native Upstream Native vllm-ascend(official plugin of the vllm-project org, version number aligned with upstream vllm;pins down the entire torch/transformers chain, see Ascend section) vLLM-MUSA(official backend, open-source fork) unverified
sglang Upstream Native unverified Officially supported (since 2025-08,--device npu --attention-backend ascend, sgl-kernel-npu repository) Merged into mainline (2026-05, source install + sgl-kernel) Unverified

Pattern:The inference/compilation ecosystem is shifting from "vendor-maintained forks" to "pluginization in official repositories"(the vllm hardware plugin mechanism, sglang being merged into the mainline). When checking versions, first look at the official repository's hardware support matrix, then look at the vendor fork's release line.

Key points and pitfalls for each platform

CUDA (baseline)

  • When you pip install torch, you get the CUDA version by default. Only compiling extensions from source requires the system toolkit; running only requires the driver.
  • The driver version just needs to be >= the wheel's CUDA runtime version (forward compatibility); the top-right corner of nvidia-smi shows the maximum CUDA version supported by the driver.
  • The container route is the least troublesome: nvidia-container-toolkit + official NGC images.

MUSA (including field-tested examples from 2026-09 S5000/MUSA 5.2)

  • SDK: 5.1.0 corresponds to CUDA 12.8 (released 2026-05); installation = official apt repository/offline package, user-space versioned directory; 5.2 package family = musa-toolkit-5-2 / musa-musart-5-2 / libmudnn3-musa-5-2 / libmthreads-compute,MCCL is packaged separately by card model(S5000 uses mccl-s5000, not the S4000 version); the pure user-space installation method does not touch mthreads-driver/dkms.
  • Exact vendor index URL:https://dl.mthreads.com/repo/api/pypi/pypi/simple; the repository contains a large number of ordinary mirror packages,being in the vendor repository ≠ must be installed from there。
  • Package routing (which ones must come from the vendor repository): torch / torch_musa / torchvision / torchaudio / triton (i.e., Triton-MUSA, the package is simply named triton) / tilelang_musa / mate / vllm_musa / flash_attn_3 / flash_mla / deep-gemm / sageattention → vendor repository; device-agnostic packages such as transformers / accelerate / datasets → public repository; pytorch3d, pytorch_sparse/scatter/cluster, some Lightning branches → Moore fork, install as needed.
  • Version iron rule: torch ↔ torch_musa must be exactly the same version; the wheel is labeled musa5.2.0 ↔ user-space 5.2 stack; cp310/cp312 ABI matching. Verified example rows (checked 2026-09): torchvision 0.24.1.post1+musa5.2.0 / torchaudio 2.9.1+musa5.2.0 / triton 3.6.0 / numpy 1.26.4.
  • Pitfalls:pip install torch will overwrite +musa version with the public CUDA version (silent failure, recheck pip list | grep torch）。
  • Code side: distributed backend="mccl"; the muDNN switch is in torch.backends.mudnn; the CUDA allocator env vars (PYTORCH_CUDA_ALLOC_CONF, etc.) have no effect on MUSA processes; tools: mthreads-gmi (smi) / musaInfo / musa_version_query / mccl_version.
  • CUDA compatibility layer: torchada (import torchada lets 99% of CUDA code run on MUSA, nccl can be mapped to MCCL) + MATE operator library (FlashAttention/FlashMLA/DeepGEMM interfaces). The complete installation/verification process and CUDA→MUSA API mapping table are in the MUSA instances of the accel trio.

Ascend CANN (field-verified on 2026-09-28: 910B2×8 / CANN 9.0.0 / aarch64 / py3.10, L1–L6 all passed)

  • Strong version binding chain (verified rows): driver 25.2.1/firmware 7.7.0.9.220 ↔ CANN 9.0.0 ↔ torch 2.10.0 ↔ torch_npu 2.10.0.post4 ↔ torchvision 0.25.0 / torchaudio 2.10.0 ↔ triton-ascend 3.2.2 ↔ vllm 0.23.0 + vllm-ascend 0.23.x ↔ transformers 5.5.4. How to check the companion table:curl -s https://raw.githubusercontent.com/Ascend/pytorch/master/README.md | grep -A2 CANN; for the pinned versions of each vllm-ascend release, check pypi.org/pypi/vllm-ascend/<ver>/json 's requires_dist.
  • Wheel status (2026-09-28): torch_npu is on public PyPI, aarch64 has covered cp310–cp313 since 2.9.x, butofficial releases occasionally ship only some ABIs(2.12.0 only has cp312/313); check the ABI against the simple page filenames, don't trust pip index 's JSON metadata (on the mirror it lags by months, stuck on an old version).
  • Five pitfalls of installing vllm-ascend(2026-09-28, all hit and all solved):
    1. Upstream vllm pins torch one notch higher than vllm-ascend pins it (vllm 0.23.0→torch 2.11, vllm-ascend 0.23.x→2.10): after installing vllm you must re-pin the whole torch family + transformers, otherwise the torch_npu ABI breaks.
    2. post versions often only have an sdist (whereas rc versions actually have full-ABI wheels): for py3.10/3.11 use the same-line rc wheel + --no-deps, and manually fill in missing dependencies per its METADATA (a whole string of xgrammar/compressed-tensors/numba/quart).
    3. triton-ascend and upstream triton share the same name and namespace and are mutually exclusive(coexisting = the latter shadows the former, symptom 'function' object is not subscriptable / No module named triton._C.libtriton.ascend; uninstalling them takes away each other's files): solution = uninstall both,--no-deps install only triton-ascend, andinstall it last。
    4. new triton-ascend versions lag in publishing to public sources (3.2.2 is only on Huawei Cloud):--find-links https://mirrors.huaweicloud.com/ascend/repos/pypi/triton-ascend/。
    5. NNAL (libatb.so) is a hard dependency of vllm, it's not in the CANN toolkit unified package (--feature only has ascendc) and there's no pip package either: when there's no official mirror channel, you can stream-extract it from the official CANN docker image (quay.io/ascend/cann:-910b-…) via the registry API /usr/local/Ascend/nnal and install it locally; at runtime you must source both: CANN set_env + nnal/atb/set_env.sh, otherwise libatb.so not found → the engine won't start.
  • torch_npu usage surface (verified):torch.npu.* mirrors the CUDA API; autocast device_type="npu"；transformers device_map="auto" can be layered across multiple NPUs; HCCL single/two-process all_reduce is exact. For 910B2, plan HBM at 61GB per card (system reserves ~3.4GB).
  • transformers is pinned from both sides, causing a conflict: vllm-ascend pins transformers (0.23.x↔5.5.4), while the model's remote code often pins an old version (openPangu↔4.53.2, which breaks in two places on 5.x: LossKwargs/rope key names): the solution in choosing your approach = a separate small environment for that model, or go through vllm (which doesn't depend on transformers remote code); don't patch symbol by symbol in the main environment (you'll never finish).
  • Symptom quick reference:ERR99999 UNKNOWN application exception is the Ascend abnormal termination line, the real cause is in the traceback above it;NPUCachingAllocator ... 32 padding size warning is harmless, don't fix it.
  • tilelang-ascend: CANN ≥ 8.3.RC1, torch_npu ≥ 2.6.0; download the wheel from GitHub Releases matching CANN/Python/architecture.
  • Hardware differences (unavoidable when writing kernels): AI Core is split into Cube/Vector; each core's UB is only 192KB (if BLOCK_SIZE is too large it overflows);out-of-bounds access has no fault tolerance and directly causes a Device Hang——the mask must be strict。
  • The official in-house inference engine MindIE coexists with vllm-ascend; choose based on the scenario (MindIE is commercial and closed-source, vllm-ascend is community open-source).

ROCm

  • The official support matrix limits the distro + kernel version, check before installing (rocm.docs.amd.com).
  • The device name is still cuda(HIP translation), CUDA code basically needs zero changes; rccl replaces nccl.
  • Upstream triton/tilelang/vllm/sglang has native support (for the version matrix see pytorch.org and each framework's release notes).

PPU (Alibaba T-Head)

  • Zhenwu 810E: GPGPU route (not ASIC), deployed at scale on Alibaba Cloud, shipping for over a year (officially confirmed 2026-01).
  • Public SDK/docs are scarce: setting up the environment doesn't rely on public sources (not finding something doesn't mean it doesn't exist); the main path = Alibaba Cloud instances/container images + Alibaba Cloud docs and tickets.
  • First contact: first check the image and runtime instructions given in the instance spec docs, and clarify "what comes preinstalled and how to use it" before starting.

MetaX MACA (quick notes)

  • Wheels carry the +metax/+maca suffix (torch 2.6.0+metax, vllm 0.8.5+maca, triton 3.0.0+metax); the anti-mixing rule is the same as MUSA: public sources pip install torch will override the adapted version.
  • Main installation path = docker images/release packages from the MetaX developer community (containers must mount /dev/mxcd /dev/dri); MACA version numbers look like mc2.32.0.3 and must strictly match the release package.
  • MACA 3.3 supports all PyTorch 2.8 operators; inference supports vllm/SGLang/Transformers.

General pitfall prevention (all platforms)

  • Version quad: driver ↔ SDK ↔ torch backend ↔ compiler fork (triton/tilelang); upgrading any one link requires rechecking the entire chain's compatibility matrix.
  • pip list | grep -E 'torch|triton|vllm' checklocal version suffix(+musa/+metax/+npu…) to determine whether it is a platform-adapted version; no suffix ≈ public version, most likely it has been overridden.
  • Not finding platform wheels on public sources is normal, not a sign that the platform is unsupported—go to the vendor docs/download center (see the research path in accel-init phase 3).
  • Visible device variables differ by platform: CUDA_VISIBLE_DEVICES / MUSA_VISIBLE_DEVICES / ASCEND_RT_VISIBLE_DEVICES / HIP_VISIBLE_DEVICES(ROCm).
  • Plugins can also override versions: after installing vllm-/sglang- and other platform plugins, immediately recheck whether torch/transformers/torchvision have been overridden and re-pin them—"anti-mixing" is not just about preventing pip install torch, but also about preventing any package that pins torch (Ascend 2026-09-28 instance: vllm bumped torch up a notch).
  • Mutually exclusive package families with the same name: vendor triton forks share the namespace with upstream triton (both MUSA/Ascend are confirmed); coexistence = silent breakage (kernel decorators stop working and become ordinary functions, backend submodules missing); the correct fix = uninstall both cleanly,--no-deps install only the vendor version, and put it last in the installation sequence.
  • The package JSON metadata of mirror sources can lag by months (measured: Tsinghua mirror's torch-npu metadata is stuck at 2.3.1 while the actual version is already 2.12): check versions/ABI based on the file names on the simple page or the pypi.org JSON.
  • "Missing runtime library type" hard dependencies often have hidden system packages (e.g., Ascend vllm with respect to NNAL/libatb.so): when you see cannot open shared object file , first think of missing system-level components, and check the Requirements table in the official installation docs instead of continuing to struggle with pip.

Evolution (this repository is ready to be updated at any time)

Positioning: the inflow point for experience = experiments and pitfalls on real servers (installed, ran successfully, errors, version mismatches are all material). Process:

  1. Real-machine experience → find the corresponding entry in this repository; if there is none, add a line in the correct table/section, do not create a new orphan section.
  2. Update rules: conclusions carry dates and evidence (command output/official documentation links);(未验证) entries are, after real-machine verification,promoted or deleted; overturned conclusions are directly rewritten to the currently correct version (git keeps the trace; outdated narratives are not retained).
  3. Structural invariants: the four-layer form of overview table / ecosystem matrix / platform sections / general pitfall prevention is maintained; a new platform = one row in the overview + one section; pitfalls specific to a single platform are not elevated to general pitfall prevention.
  4. Updates can be made directly on the local copy on the server machine (the place where evolution happens = the real machine). After editing, follow the fixed loop: download to the development machine → merge into the repository and push → reinstall on the local machine to pull the new version. Pitfall: before local modifications have flowed back, do not rerun setup's kb overwrite refresh (it will wipe them out).