You are an engineer focused on migrating PyTorch code to Ascend NPU. This Skill provides the specific operational steps, replacement rules, and verification methods for code migration.
Official reference documentation:
- Training migration guide:https://www.hiascend.com/document/detail/zh/Pytorch/2600/ptmoddevg/trainingmigrguide/FrameworkPTAdapter/26.0.0/zh/pytorch_model_migration_fine_tuning/mig_methods_comp.md
- Affinity API replacement (performance tuning):https://www.hiascend.com/document/detail/zh/Pytorch/2600/ptmoddevg/trainingmigrguide/FrameworkPTAdapter/26.0.0/zh/pytorch_model_migration_fine_tuning/affinity_api_repl.md
- PyTorch native API support level:https://www.hiascend.com/document/detail/zh/Pytorch/2600/apiref/PyTorchNativeapi/docs/zh/native_apis/pytorch_2-10-0/overview.md
- torch_npu extension APIs:https://www.hiascend.com/document/detail/zh/Pytorch/2600/apiref/Extensionapi/docs/zh/extension_apis/overview.md
- torch_npu official repository:https://gitcode.com/Ascend/pytorch
⚠ Versions are continuously updated; the version number in the URL (e.g., 26.0.0) iterates with the PyTorch framework adaptation version. You MUST replace the version number in the URL with the version number actually in use and access the documentation for the corresponding version.
I. Migration method selection
Ascend provides two migration methods:
1.1 Automatic migration (recommended)
Through transfer_to_npu , automatically map runtime interfaces such as torch.cuda to the corresponding torch.npu interfaces, without manually modifying the code.
import torch
import torch_npu
from torch_npu.contrib import transfer_to_npu
Applicable scenarios: Simple scenarios that do not use advanced CUDA capabilities (such as custom operators, direct GPU memory operations, etc.)
Limitations:
- Only supports PyTorch 1.8.1 and above
- Automatic migration is suitable for simple scenarios that do not use advanced CUDA capabilities. If custom operators, active GPU memory allocation, or other operations are involved, additional manual migration adaptation is required
1.2 Manual migration
Manually modify CUDA-related calls in the code to the corresponding NPU interfaces. Suitable for scenarios that automatic migration cannot cover.
Prerequisites:
- The training task code to be migrated has been trained multiple times on GPU with stable convergence
- Migration environment preparation has been completed
Constraints and limitations:
- After installing the plugin, most capabilities can correspond to their use on GPU, but not all behaviors correspond one-to-one with those on GPU
- Under torch_npu, when the PyTorch version is lower than 2.1.0, a process can operate only one Ascend card, and the ability for one process to operate multiple cards is not supported; only in PyTorch 2.1.0 and above does torch_npu support using multiple Ascend cards in one process
- There are very many third-party development libraries based on PyTorch, such as transformers, accelerate, deepspeed, and Megatron-LM. Ascend has also made adaptation plugin libraries similar to PyTorch Adapter for these third-party libraries, which can be used as needed from the official gitcode.com/Ascend repository. Some third-party libraries, such as the latest version of deepspeed, already natively support NPU
II. Manual migration: device mapping replacement
2.1 torch_npu initialization
After torch_npu is installed, this part is not directly implanted into PyTorch to take effect; the user needs to call it explicitly.
import torch_npu
After the call, the frontend injects it into the torch object via monkey-patching, and the backend registers the NPU device and HCCL's parameter-plane communication capabilities.
2.2 Basic device replacement rules
| Original code | Replace with | Description |
|---|---|---|
torch.device('cuda:0') |
torch.device('npu:0') |
Device specification |
torch.cuda.set_device(0) |
torch.npu.set_device(0) |
Set the current device |
torch.cuda.is_available() |
torch.npu.is_available() |
Device availability check |
torch.cuda.device_count() |
torch.npu.device_count() |
Device count query |
torch.cuda.current_device() |
torch.npu.current_device() |
Get the current device |
.cuda() |
.npu() |
Tensor/Model device transfer |
.to('cuda:0') |
.to('npu:0') |
Tensor device transfer |
torch.cuda.synchronize() |
torch.npu.synchronize() |
Device synchronization |
torch.cuda.memory_allocated() |
torch.npu.memory_allocated() |
Memory query |
torch.cuda.max_memory_allocated() |
torch.npu.max_memory_allocated() |
Peak memory query |
torch.cuda.empty_cache() |
torch.npu.empty_cache() |
Memory cache clearing |
2.3 Mixed precision replacement rules
When adapting a model from GPU to NPU, the code torch.cuda.amp needs to be changed to torch_npu.npu.amp.
| Original code | Replace with |
|---|---|
torch.cuda.amp.autocast() |
torch.npu.amp.autocast() |
torch.cuda.amp.GradScaler() |
torch.npu.amp.GradScaler() |
The Cube compute unit of the Ascend NPU supports only FP16 matrix operations; FP32 matrix operations cannot invoke the Cube's high computing power. Using FP16 across the entire network has a small numerical range and easily leads to vanishing gradients, so mixed precision training (AMP) is a necessary operation on the NPU, not an optional one.
2.4 Distributed Training Replacement Rules
Ascend supports only DDP mode, not DP mode.
| Original code | Replace with | Description |
|---|---|---|
backend="nccl" |
backend="hccl" |
Collective communication backend |
torch.nn.DataParallel |
torch.nn.parallel.DistributedDataParallel |
NPU does not support DP; MUST be changed to DDP |
torch.cuda.default_generators |
torch_npu.npu.default_generators |
Random number generator |
Distributed initialization method:
# 方式1:通过接口设置
dist.init_process_group(
backend='hccl',
init_method="tcp://127.0.0.1:29688",
world_size=args.world_size,
rank=args.rank
)
# 方式2:通过环境变量设置
os.environ['MASTER_ADDR'] = '127.0.0.1'
os.environ['MASTER_PORT'] = '29688'
dist.init_process_group(
backend='hccl',
world_size=args.world_size,
rank=args.rank
)
2.5 Interfaces Not Supported by NPU
The following interfaces are not supported on NPU or behave differently; they MUST be handled manually:
torch.nn.DataParallel: not supported, MUST be changed to DDPamp_Cmodule: not supported, MUST be manually deletedtorch.cuda.get_device_capability: after migration returnsNone, if an error occurs it needs to be manually changed to a fixed valuetorch.cuda.get_device_properties: the return value does not include theminorandmajorattributes; the related code needs to be commented out
III. Equivalent Interface Replacement
3.1 Replacement Process
- Filter out the interfaces not supported by NPU from the interface analysis report
- Consult the official Ascend interface support documentation to confirm whether the interface has been adapted:
- PyTorch native interface support:https://www.hiascend.com/document/detail/zh/Pytorch/2600/apiref/PyTorchNativeapi/docs/zh/native_apis/pytorch_2-10-0/overview.md
- torch_npu extension interfaces:https://www.hiascend.com/document/detail/zh/Pytorch/2600/apiref/Extensionapi/docs/zh/extension_apis/overview.md
- If there is no official adaptation solution, look for alternatives in the following priority order:
- Priority 1: Use the extension interfaces provided by torch_npu
- Priority 2: Use an equivalent combination of PyTorch native interfaces
- Priority 3: Use a custom implementation (MUST justify mathematical consistency)
3.2 Interface Support Query
Ascend officially provides interface support lists differentiated by PyTorch version; you MUST consult the corresponding documentation according to the PyTorch version actually in use:
| PyTorch version | Interface documentation path |
|---|---|
| 2.10.0 | apiref/PyTorchNativeapi/docs/zh/native_apis/pytorch_2-10-0/overview.md |
| 2.9.0 | apiref/PyTorchNativeapi/docs/zh/native_apis/pytorch_2-9-0/overview.md |
| 2.8.0 | apiref/PyTorchNativeapi/docs/zh/native_apis/pytorch_2-8-0/overview.md |
| 2.7.1 | apiref/PyTorchNativeapi/docs/zh/native_apis/pytorch_2-7-1/overview.md |
| 2.6.0 | apiref/PyTorchNativeapi/docs/zh/native_apis/pytorch_2-6-0/overview.md |
| 2.1.0 | apiref/PyTorchNativeapi/docs/zh/native_apis/pytorch_2-1-0/overview.md |
All the above paths are based under the documentation root directory
https://www.hiascend.com/document/detail/zh/Pytorch/2600/. Versions iterate quickly, so you MUST refer to the latest version of the documentation on the official website.
Query method:
- Determine the PyTorch version used by the current project
- Open the interface support documentation for the corresponding version
- Search for the target interface name and check whether it is in the support list
- If it is in the support list, confirm the description of behavioral differences for the interface
- If it is not in the support list, check the torch_npu extension interface documentation for an alternative
3.3 Mathematical Consistency Justification Template
For each equivalent replacement, you MUST justify it according to the following template:
### 接口替换:[原始接口] → [替换接口]
**数学定义**:
- 原始接口:y = f(x),数学定义式为...
- 替换接口:y = g(x),数学定义式为...
**等价性证明**:
- 当满足条件...时,f(x) = g(x)
- 证明过程:...
**数值验证**:
- 测试输入:生成N组随机输入
- 最大绝对误差:{value}
- 最大相对误差:{value}
- 结论:误差在可接受范围内 / 需要进一步处理
IV. Debugging Techniques
4.1 Locating Unsupported Interfaces
try:
output = model(input_npu)
except RuntimeError as e:
if "not implemented for" in str(e) and "NPU" in str(e):
print(f"不支持的接口: {e}")
4.2 Layer-by-Layer Verification
def verify_layer_by_layer(model, sample_input, device="npu"):
x = sample_input.to(device)
for name, layer in model.named_children():
x = layer(x)
print(f"Layer {name}: output shape={x.shape}, "
f"mean={x.mean().item():.6f}, std={x.std().item():.6f}")
4.3 Precision Comparison Tools
def compare_outputs(cpu_output, npu_output, atol=1e-5, rtol=1e-3):
cpu_np = cpu_output.detach().cpu().numpy()
npu_np = npu_output.detach().cpu().numpy()
max_abs_diff = abs(cpu_np - npu_np).max()
max_rel_diff = (abs(cpu_np - npu_np) / (abs(cpu_np) + 1e-8)).max()
print(f"最大绝对误差: {max_abs_diff:.2e}")
print(f"最大相对误差: {max_rel_diff:.2e}")
assert max_abs_diff < atol or max_rel_diff < rtol, "精度超出允许范围"
V. NPU Affinity Tuning
After completing the basic migration (device mapping, interface replacement), the model can already run normally on the NPU. However, to fully leverage NPU hardware performance, affinity tuning is required. The following methods come from the official Ascend documentation: https://www.hiascend.com/document/detail/zh/Pytorch/2600/ptmoddevg/trainingmigrguide/FrameworkPTAdapter/26.0.0/zh/pytorch_model_migration_fine_tuning/affinity_api_repl.md
⚠ Versions are continuously updated; the affinity API list and usage may change with versions. You MUST consult the above official website links in real time to obtain the latest information.
5.1 Overview of Affinity API Replacement
After completing the basic migration, replacing some native PyTorch interfaces in the code with Ascend NPU affinity interfaces can further improve training efficiency. The affinity interfaces are deeply optimized for the NPU hardware architecture, and can significantly reduce operator scheduling overhead and improve memory access efficiency.
5.2 Fused transpose operation:npu_confusion_transpose
torch_npu.npu_confusion_transpose used to fuse transpose + view(or reshape) combined operations, reducing the generation of intermediate tensors, and lowering memory usage and operator scheduling overhead.
Usage scenario: when the pattern of transpose immediately followed by view/reshape frequently appears in the model code (common in modules such as multi-head attention), using this API can improve performance.
# torch原生写法
import torch
data = torch.rand(64, 3, 64, 128).npu()
result = data.transpose(1, 2).view(64, 64, -1)
# torch_npu亲和写法
import torch_npu
data = torch.rand(64, 3, 64, 128).npu()
result = torch_npu.npu_confusion_transpose(data, perm=[0, 2, 1, 3], shape=[64, 64, -1])
5.3 NPU affinity optimizer
Use the fused optimizer provided by torch_npu to replace the native PyTorch optimizer. The NPU affinity optimizer reduces the overhead of multiple kernel launches during parameter updates through operator fusion.
Prerequisite: before replacing the optimizer, ensure that the import of the NPU affinity optimizer has been completed.
Common NPU affinity optimizers(taking PyTorch 2.6.0 as an example; newer versions of the documentation may add more):
| Native PyTorch optimizer | NPU affinity optimizer | Description |
|---|---|---|
torch.optim.SGD |
torch_npu.optim.NpuFusedSGD |
Fused SGD optimizer |
torch.optim.AdamW |
torch_npu.optim.NpuFusedAdamW |
Fused AdamW optimizer |
torch.optim.Adam |
torch_npu.optim.NpuFusedAdam |
Fused Adam optimizer |
# 原生PyTorch写法
import torch
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
# torch_npu亲和写法
import torch_npu
optimizer = torch_npu.optim.NpuFusedAdamW(model.parameters(), lr=1e-4)
Precautions:
- Not all optimizers have NPU fused versions; you MUST consult the official documentation to confirm the list of fused optimizers supported by the current version
- The hyperparameters of fused optimizers are basically the same as those of native optimizers, but some special parameters may differ; confirm against the official documentation
5.4 NPU affinity gradient clipping
Before replacing with the NPU affinity gradient clipping API, you MUST ensure that the NPU affinity optimizer has been used (see Section 5.3). Used together with the NPU affinity optimizer, NPU affinity gradient clipping can further reduce the number of kernel launches for parameter updates.
# 原生PyTorch写法
import torch
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
# ... loss计算 ...
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
# torch_npu亲和写法
import torch_npu
optimizer = torch_npu.optim.NpuFusedAdamW(model.parameters(), lr=1e-4)
# ... loss计算 ...
loss.backward()
torch_npu.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
Note: the specific interface name of the gradient clipping API may change with the version; you MUST consult the official documentation of the current version to confirm.
5.5 Multi-stream parallel optimization (supplementary)
Ascend NPU supports multi-stream parallel execution, and concurrency can be improved by explicitly managing compute streams:
# 创建多个stream并行执行不依赖的操作
stream1 = torch.npu.Stream()
stream2 = torch.npu.Stream()
with torch.npu.stream(stream1):
output1 = model_part1(data1)
with torch.npu.stream(stream2):
output2 = model_part2(data2)
torch.npu.synchronize() # 等待所有stream完成
Multi-stream parallelism is suitable for independent computation branches in the model, and can effectively hide data transfer latency. Before use, ensure that there are no data dependencies between the branches.
5.6 Performance tuning reference path
Affinity tuning is part of the Ascend performance tuning system. For the complete performance tuning methods and the latest affinity API list, you MUST consult the official Ascend documentation:
- PyTorch training model migration and tuning overview:https://www.hiascend.com/document/detail/zh/Pytorch/2600/ptmoddevg/trainingmigrguide/FrameworkPTAdapter/26.0.0/zh/pytorch_model_migration_fine_tuning/mig_methods_comp.md
- Affinity API replacement (including the latest API list):https://www.hiascend.com/document/detail/zh/Pytorch/2600/ptmoddevg/trainingmigrguide/FrameworkPTAdapter/26.0.0/zh/pytorch_model_migration_fine_tuning/affinity_api_repl.md
- Performance tuning methods overview:https://www.hiascend.com/document/detail/zh/Pytorch/2600/ptmoddevg/trainingmigrguide/FrameworkPTAdapter/26.0.0/zh/pytorch_model_migration_fine_tuning/npu_suit_optimize.md
⚠ The version numbers in the above URLs (such as 26.0.0) correspond to the currently adapted PyTorch framework version. Versions are continuously updated; you MUST replace the version number in the URL with the version number actually in use to obtain matching documentation content.