SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsfault_diagnose

fault_diagnose

General & other

View repository

Updated 10-15 19:58

About this skill

Fault Diagnosis

Function Overview

The Ascend fault diagnosis tool is used to detect and analyze NPU-related faults, helping users quickly locate and resolve issues.

Prerequisites

  1. Install ascend-fault-diagnose: use the ascend-fd version command to check whether it is installed
  2. Installation Guide：https://www.hiascend.com/document/detail/zh/mindcluster/730/faultdiag/faultdiagug/mindxdlFDUG008.html

Instructions

Usage Recommendations

  1. Cluster specifications limit: due to the Linux system maximum process count limit (default 1024), the cluster specification is recommended to be ≤128 servers (1024 cards). If the number of servers exceeds this specification, you need to use the ulimit -n command to adjust the file descriptor upper limit
  2. Avoid pipe commands: when using diagnosis tool commands, try not to use pipe commands, as this may affect the retrieval of user IPs and log auditing

Supported Scenarios

  1. Full-machine full-card training and inference: fault diagnosis capabilities are only supported for full-machine full-card training and inference tasks; non-full-card scenarios may cause incorrect or failed fault root cause localization
  2. IPv4 environment: currently only IPv4 is supported, IPv6 is not supported

System Time Description

  1. Synchronize server time: please synchronize the system time of each training and inference server; inconsistent system time may cause inaccurate analysis results
  2. Synchronize Host and Device time: please synchronize the Host system time and the Device system time on each server
  3. Synchronize container time: if using containers to execute tasks, please synchronize the time between the host machine and the container

Usage Restrictions

Log Version Compatibility Table

Log File Corresponding Software Software Version Description
CANN application logs CANN 7.0.RC1 and later Host-side and Device-side application logs
PyTorch framework training and inference logs PyTorch1.11.0 framework adaptation plugin 5.0.RC3 and later -
MindSpore framework training logs MindSpore 2.1.0 and later Some fault types include version descriptions
TensorFlow framework training logs TensorFlow - Only user-defined TensorFlow faults are supported
Host OS logs - - Supports CentOS 7.6, Debian 10.0, EulerOS 2.10/2.12, CTyunOS 22.06, etc.; log size is recommended to be ≤512MB
Device-side logs Ascend HDK 23.0.RC3 and later -
MindCluster component logs Device Plugin/NodeD, etc. 6.0.RC3 and later -
MindIE component logs MindIE Server/LLM, etc. 6.0.0 and later -
AMCT component logs AMCT 7.0.RC1 and later Released as integrated in the CANN package
MindIE Pod console logs MindIE Pod - -

Diagnosis Process

日志采集 → 日志清洗 → 故障诊断

Information to be provided before starting diagnosis

Before starting fault diagnosis, the user needs to provide the following information:

  1. Fault occurrence stage: The stage of training and inference (before/during/after)

    • Before: Before training and inference start, collect NPU environment check files
    • During: During training and inference, collect NPU status monitoring, host resource information, etc.
    • After: After training and inference end, collect complete logs for diagnosis

  2. Node connection data: Provide connection information for all nodes, used for log collection and cleaning

    • Node IP address list
    • SSH connection information (username, password, or key)
    • Node role (master/worker)

Log collection script

Use the log_collector.py script to automatically collect logs:

Node configuration file format（JSON）：

[
    {"ip": "192.168.1.1", "user": "root", "password": "xxx", "name": "master"},
    {"ip": "192.168.1.2", "user": "root", "key_file": "/path/to/key", "name": "worker-1"}
]

Usage：

# 训练前采集
python3 log_collector.py -s before -n nodes.json -o ./logs

# 训练中采集
python3 log_collector.py -s during -n nodes.json -o ./logs

# 训练后采集
python3 log_collector.py -s after -n nodes.json -o ./logs

Parameter description：

Parameter Description
-s, --stage Collection stage: before/during/after
-n, --nodes Node configuration file path
-o, --output Output directory

I. Log collection

1.1 Log collection before training and inference

File：npu_info_before.txt
Path：采集目录/environment_check/

Collection command：

Tool Command Description
hccn_tool -net_health -g Network health status
hccn_tool -link -g RoCE link status
hccn_tool -ip -g IP and mask
hccn_tool -stat -g Packet sending and receiving statistics
hccn_tool -link_stat -g Historical link statistics
npu-smi info Device basic information
npu-smi info -t ecc ECC count
npu-smi info -t board Hardware information
npu-smi info -t usages Memory usage
npu-smi info -c 0 -t health Chip health status

1.2 Log collection during training and inference

File type Naming constraint Collection method
NPU network port statistics npu_{id}_details.csv hccn_tool -stat, every 15 seconds
NPU status monitoring npu_smi_{id}_details.csv npu-smi, every 15 seconds
Host resource information host_metrics_{core_num}.json top command
MindIE Pod logs {pod_name}.json kubectl logs

1.3 Log collection after training and inference

Log type Naming constraint Storage path
NPU environment check npu_info_after.txt environment_check/
User training logs rank-*.log、worker-*.log Collection directory
CANN application logs plog-*.log、device-*.log process_log/
Host-side logs messages、dmesg Collection directory
Device-side logs device-os_*.log device_log/
MindCluster logs devicePlugin*.log etc. dl_log/
MindIE logs mindie-*.log mindie/log/debug/
AMCT logs amct_*.log amct_log/

Host-side log collection command：

# 内核消息日志
dmesg -T | tail -n 100000 > 采集目录/dmesg

# 系统监测日志
cp /var/log/sysmonitor.log 采集目录/

# Device侧日志
msnpureport

MindCluster log collection command：

cp /var/log/mindx-dl/devicePlugin 采集目录/dl_log
cp /var/log/mindx-dl/noded 采集目录/dl_log
cp /var/log/ascend-docker-runtime 采集目录/dl_log
cp /var/log/mindx-dl/volcano-scheduler 采集目录/dl_log
cp /var/log/mindx-dl/volcano-controller 采集目录/dl_log
cp /var/log/mindx-dl/npu-exporter 采集目录/dl_log

1.4 Collection directory structure

采集目录/
├── messages                    # 主机侧操作系统日志
├── dmesg                       # 主机侧内核消息日志
├── sysmonitor.log              # 主机侧系统监测日志
├── rank-*.txt                  # 训练控制台日志
├── process_log/                # CANN应用日志
├── device_log/                 # Device侧日志
├── dl_log/                     # MindCluster组件日志
├── mindie/                     # MindIE组件日志
├── amct_log/                   # AMCT组件日志
└── environment_check/          # NPU环境检查文件
    ├── npu_info_before.txt
    ├── npu_info_after.txt
    ├── npu_smi_*_details.csv
    ├── npu_*_details.csv
    └── host_metrics_*.json

II. Log cleaning

2.1 Environment configuration

# 配置环境变量（普通用户）
export PATH=$PATH:/usr/local/python3.7.5/bin

# 验证配置
ascend-fd -h

2.2 Execute cleaning

# 创建输出目录
mkdir 清洗输出目录

# 执行清洗命令
ascend-fd parse -i 采集目录 -o 清洗输出目录 --performance

2.3 Cleaning output files

File Description
ascend-kg-parser.json Fault event analysis cleaning result (inference engine input)
ascend-kg-analyzer.json Fault event analysis cleaning result
ascend-rc-parser.json Root cause node analysis cleaning result
device_ip_info.json Device IP information
nic_clean.csv Network congestion cleaning results
nad_clean.csv Frequency reduction cleaning results
process_*.csv CPU resource preemption cleaning results
plog-parser-*.log Root cause node analysis cleaned logs

2.4 Cleaning service flow logs

Clean service flow logs through the Python API interface.

Import interface：

from ascend_fd import parse_fault_type

Invocation example：

results, err_msg_list = parse_fault_type(input_log_list)

Input parameter input_log_list format：

[
    {
        "log_domain": {
            "server": "10.1.1.1",
            "device": ["0", "1"]
        },
        "log_items": [
            {
                "item_type": "MindIE",
                "log_lines": [
                    "[ERROR] xxx",
                    "[ERROR] yyy"
                ]
            }
        ]
    }
]

Input parameter description：

Field Parameter Type Required Description
log_domain Dictionary Yes Log domain
server String Yes Server address
device List Yes Full card information of all cards that have experienced faults
log_items List Yes Log item
item_type String Yes Log type
log_lines List Yes Log line to be parsed

Output parameter results format：

[
    {
        "error_type": "AISW_MindIE_MS_HttpServer_01",
        "fault_domain": "Software",
        "attribute": {
            "key_info": "",
            "component": "MindIE",
            "module": "MS",
            "cause": "Httpserver通信超时",
            "description": "等待时间超过设定的时延。",
            "suggestion": ["1. 请联系华为工程师处理；"]
        },
        "device_list": [
            {
                "server": "172.0.0.1",
                "device": ["0", "1", "2"]
            }
        ]
    }
]

Output parameter description：

Field Parameter Type Description
error_type String Fault code
fault_domain String Fault domain
attribute Dictionary Fault attribute
key_info String Key log
component String Fault component
module String Fault module
cause String Fault cause
description String Fault description
suggestion List Recommended solution
device_list List List of devices where the fault occurred

Error information err_msg_list: List of error information generated during interface execution.

2.5 Root cause node cleaning and diagnosis

Clean and diagnose root cause nodes through the Python API interface.

Import interface：

from ascend_fd import parse_root_cluster
from ascend_fd import diag_root_cluster

Invocation example：

# 根因节点清洗
rc_parse_results, rc_parse_err_msg = parse_root_cluster(input_log_list)

# 根因节点诊断
results, err_msg_list = diag_root_cluster(rc_parse_results)

Input parameter input_log_list format：

[
    {
        "log_domain": {
            "server": "10.1.1.1",
            "instance_id": "instance_name"
        },
        "log_items": [
            {
                "item_type": "plog",
                "pid": 3199,
                "device_id": 0,
                "rank_id": 0,
                "log_lines": ["[ERROR] xxx."]
            }
        ]
    }
]

Input parameter description：

Field Parameter Type Required Description
log_domain Dictionary Yes Log domain
server String Yes Server address
instance_id String Yes Instance ID
log_items List Yes Log item
item_type String Yes Log type
pid Int Yes Process ID
device_id Int Yes Device ID
rank_id Int Yes Rank ID
log_lines List Yes Log line to be parsed

Output parameter results format：

{
    'analyze_success': True,
    'fault_description': {
        'code': 102,
        'string': '所有有效节点的Plog都没有错误日志信息，无法定位根因节点。'
    },
    'root_cause_device': ['ALL Device'],
    'device_link': [],
    'remote_link': '',
    'first_error_device': '',
    'last_error_device': ''
}

Output parameter description：

Field Parameter Type Description
analyze_success Bool Whether diagnosis is successful (True/False)
fault_description Dictionary Fault description
code Int Fault code
string String Fault code description
root_cause_device List Root cause device information
device_link List Root cause node chain
remote_link String Inter-card wait chain
first_error_device String Device where the error occurred earliest
last_error_device String Device where the error occurred latest

Error information：

  • rc_parse_err_msg: List of error information generated during the cleaning process
  • err_msg_list: List of error information generated during the diagnosis process

2.6 Fault event cleaning and diagnosis

Clean and diagnose fault events through the Python API.

Import API：

from ascend_fd import parse_knowledge_graph
from ascend_fd import diag_knowledge_graph

Call Example：

# 故障事件清洗
kg_parse_results, kg_parse_err_msg = parse_knowledge_graph(input_log_list, custom_entity)

# 故障事件诊断
results, err_msg_list = diag_knowledge_graph(kg_parse_results)

Input parameter input_log_list format：

[
    {
        "log_domain": {
            "server": "10.1.1.1"
        },
        "log_items": [
            {
                "item_type": "MindIE",
                "path": "/log/debug/mindie-ms_11_202411061400.log",
                "device_id": 0,
                "modification_time": "2025-08-21 23:50:59.999999",
                "component": "Controller",
                "log_lines": ["[ERROR] xxx."]
            }
        ]
    }
]

Input parameter description：

Field Parameter Type Required Description
log_domain Dictionary Yes Log domain
server String Yes Server address
log_items List Yes Log entry
item_type String Yes Log type
path String No Log file path (required when cleaning NPU environment check files)
device_id Int No Device card number
modification_time String No Log modification time (used as the fault occurrence time)
component String No Component name (supports Coordinator and Controller)
log_lines List Yes Log line to be parsed

Custom fault entity custom_entity format：

{
    "41001": {  # 故障码（自定义，不能与已有故障码相同）
        "attribute.class": "Software",
        "attribute.component": "AI Framework",
        "attribute.module": "Compiler",
        "attribute.cause_zh": "抽象类型合并失败",
        "attribute.description_zh": "对函数输出求梯度时，抽象类型不匹配，导致抽象类型合并失败。",
        "attribute.suggestion_zh": [
            "1. 检查求梯度的函数的输出类型与sens_param的类型是否相同，如果不相同，修改为相同类型；",
            "2. 自动求导报错Type Join Failed"
        ],
        "attribute.error_case": [
            "grad = ops.GradOperation(sens_param=True)",
            "# test_net输出类型为tuple(Tensor, Tensor)",
            "def test_net(a, b):",
            "    return a, b"
        ],
        "attribute.fixed_case": [
            "grad = ops.GradOperation(sens_param=True)",
            "# test_net输出类型为tuple(Tensor, Tensor)",
            "def test_net(a, b):",
            "    return a, b"
        ],
        "rule": [{"dst_code": "20106"}],
        "source_file": "TrainLog",
        "regex.in": ["Abstract type", "cannot join with"]
    }
}

custom_entity parameter description：

Field Parameter Type Required Description
attribute.class String Yes Fault category
attribute.component String Yes Fault component
attribute.module String Yes Fault module
attribute.cause_zh String Yes Fault cause
attribute.description_zh String Yes Fault description
attribute.suggestion_zh List Yes Suggested solution list
attribute.error_case List No Error example code
attribute.fixed_case List No Fix example code
rule List Yes Fault propagation rule
source_file String Yes Log source file type
regex.in List Yes Match regular expression list

Output parameter results format：

[
    {
        'analyze_success': True,
        'version_info': {},
        'note': '',
        'fault': [{
            'code': 'NORMAL_OR_UNSUPPORTED',
            'component': '',
            'module': '',
            'cause_zh': '故障事件分析模块无结果',
            'description_zh': '可能为正常训练作业，无故障发生。',
            'suggestion_zh': '1. 若存在问题无法解决，请联系华为工程师处理',
            'class': '',
            'fault_source': ['1.1.1.1 device-Unknown'],
            'fault_chains': []
        }]
    }
]

Output parameter description：

Field Parameter Type Description
analyze_success Bool Whether diagnosis is successful (True/False)
version_info Dictionary Version information
note String Remarks
fault List Fault event list
code String Fault code
component String Fault component
module String Fault module
cause_zh String Fault cause
description_zh String Fault description
suggestion_zh String Suggested solution
class String Fault category
fault_source List Fault source
fault_chains List Fault propagation chain

Error information：

  • kg_parse_err_msg: List of error information generated during the cleaning process
  • err_msg_list: List of error information generated during the diagnosis process

III. Fault Diagnosis

3.1 Cluster Fault Diagnosis

# 创建诊断结果输出目录
mkdir 诊断结果输出目录

# 执行诊断命令（默认仅诊断根因节点、故障事件两个模块）
ascend-fd diag -i 诊断输入目录 -o 诊断结果输出目录

# 执行完整诊断（包含性能劣化检测模块）
ascend-fd diag -i 诊断输入目录 -o 诊断结果输出目录 --performance

3.2 Standalone Fault Diagnosis

Standalone diagnosis returns the corresponding data of the fault event module by default, which is suitable for single-node fault troubleshooting.

# 创建单机诊断结果输出目录
mkdir 单机诊断结果输出目录

# 执行单机诊断命令
ascend-fd single-diag -i 采集目录 -o 单机诊断结果输出目录

Standalone diagnosis output directory：

单机诊断结果输出目录/
└── fault_diag_result/
    └── diag_report.json    # 诊断结果

Standalone diagnosis description：

  • Standalone diagnosis scans the fault events of all valid logs in the node
  • If fault event analysis appears in the output, it indicates that the current fault may cause abnormal exit of training or inference tasks
  • When an execution error occurs, you can view all exception information through the diag_report.json file

3.3 Super Node Fault Diagnosis

Super node fault diagnosis provides three scenarios:

  • Super node topology information non-manual association scenario
  • Super node topology information manual association scenario
  • Missing Host log scenario

3.3.1 Non-manual association scenario

Requires that all three types of logs—BMC, Host, and LCNE—exist simultaneously; none of the log types may be missing.

# 创建超节点诊断结果输出目录
mkdir 超节点诊断结果输出目录

# 执行超节点诊断命令
ascend-fd diag -i 诊断输入目录 -o 诊断结果输出目录 -s super_pod

Cleaning result directory structure：

超节点清洗结果输出目录/
├── bmc/
│   ├── bmc_xxx.xx.xx.xx4_1/
│   │   ├── ascend-kg-analyzer.json
│   │   ├── ascend-kg-parser.json
│   │   └── server-info.json
│   └── bmc_xxx.xx.xx.xx5_1/
│       └── ...
├── host/
│   ├── log_collect_node-29-121_20250616/
│   │   ├── ascend-kg-analyzer.json
│   │   ├── ascend-kg-parser.json
│   │   ├── ascend-rc-parser.json
│   │   └── server-info.json
│   └── log_collect_node-29-124_20250616/
│       └── ...
└── lcne/
    ├── xxx.xx.xx.xx6/
    │   └── ...
    └── xxx.xx.xx.xx7/
        └── ...

Diagnosis result output：

fault_diag_result/
├── diag_report.json    # 诊断结果
└── topo_info.json      # 超节点拓扑信息

3.3.2 Manual association scenario

During cleaning, the three types of logs—BMC, Host, and LCNE—need to be manually associated.

Cleaning command example：

ascend-fd parse \
  --host_log parse_input/host/xxx.xx.xx.131/host_log/ \
  --mindie_log parse_input/host/xxx.xx.xx.131/mindie/ \
  --process_log parse_input/host/xxx.xx.xx.131/process_log/ \
  --bmc_log parse_input/bmc/worker-104 \
  --lcne_log parse_input/lcne/worker-204 \
  -o 清洗结果输出目录/worker-1

Cleaning result directory structure：

超节点清洗结果输出目录/
├── worker-1/
│   ├── ascend-kg-analyzer.json
│   ├── ascend-kg-parser.json
│   ├── ascend-rc-parser.json
│   ├── device_ip_info.json
│   └── server-info.json
├── worker-2/
│   └── ...
└── worker-5/
    └── ...

Diagnosis command：

ascend-fd diag -i 诊断输入目录 -o 诊断结果输出目录

3.3.3 Super node-specific fault types

Fault name Description Recommended solution
Forwarding engine module function failure LANSWITCH chip unstable Contact Huawei engineers for handling
Forwarding chip port degraded to 1/2 lane fault Forwarding chip port L1<-->CPU fault Contact Huawei engineers for handling
Forwarding chip port down fault Forwarding chip port down fault L1<-->CPU Contact Huawei engineers for handling
Forwarding engine partial function failure Forwarding chip configuration error alarm Contact Huawei engineers for handling
Forwarding engine overall function failure Forwarding chip internal fatal fault Contact Huawei engineers for handling
MindIE link establishment failure MindIE instance link establishment failure Troubleshoot the node where the link establishment failure occurred

3.3.4 Missing Host log scenario

When Host logs are missing, the BMC and LCNE cleaning results need to be stored in the same directory.

Cleaning result directory structure(LCNE logs only):

超节点清洗结果输出目录/lcne/
├── worker-200/
│   ├── ascend-kg-analyzer.json
│   ├── ascend-kg-parser.json
│   └── server-info.json
├── worker-201/
│   └── ...
├── worker-202/
│   └── ...
├── worker-203/
│   └── ...
└── worker-204/
    └── ...

Diagnosis command：

ascend-fd diag -i 诊断输入目录/lcne -o 诊断结果输出目录

Diagnosis result output：

fault_diag_result/
└── diag_report.json    # 诊断结果

Notes：

  • When Host logs are missing, root cause node analysis cannot locate the root cause node (displays Unknown Device）
  • The phenomenon description will prompt "No valid Plog file found"
  • Fault event analysis can still detect LCNE/BMC-related faults

3.3.5 Precautions

  • If a certain type of LCNE or BMC log is missing, use the manual association scenario
  • If Host logs are missing, store the cleaning output results in the same folder for diagnosis
  • When the log level is configured too low, log flooding may flush out key logs, making diagnosis impossible

3.4 Diagnosis report interpretation

The diagnosis report contains the following modules:

Module Description
Version information Fault-Diag version number
Root cause node analysis Locate the device node where the fault occurred
Fault event analysis Identify the specific fault type and recommended solution
Device resource analysis Analyze device resource usage (requires the --performance parameter)
Network congestion analysis Analyze the network status between nodes (requires the --performance parameter)

Key parameter description

Module Parameter Description
Root cause node analysis Root cause node The Device where the root cause device is located
Phenomenon description Phenomenon description of the root cause node analysis
First error node The Device in the task where the error occurred earliest
Last error node The Device in the task where the error occurred latest
Root cause node chain Propagation relationship of faulty nodes when retransmission exceeds the limit
Inter-card wait chain Propagation relationship of faulty nodes when Socket/Notify timeout occurs
Fault event analysis Status code Fault code or NORMAL_OR_UNSUPPORTED
Fault name Specific fault name
Fault category Fault category and the component and module it belongs to
Faulty device The device where the fault occurred
Fault description Detailed description of the fault
Recommended solution Handling suggestions for the fault
Key logs The fault logs corresponding to the fault
Key propagation chain The longest chain in the fault causation relationship
Device resource analysis Status code Fault code or NODE_DIAGNOSIS_NORMAL
Faulty device Name of the node where the fault occurred
Faulty process PID of the process where the fault occurred
Fault interval Time interval and probability of the fault occurrence
Fault name Specific fault name
Network congestion analysis Status code Fault code or NET_DIAGNOSIS_NORMAL
Faulty device Name of the node where the fault occurred
Faulty node List of Devices where faults occurred
Fault name Specific fault name

3.5 Common faults and recommended solutions

Fault name Description Recommended solution
Link Down: NPU-side intermittent disconnection error A Link Down intermittent disconnection error occurs on the NPU network port, and the disconnection lasts more than 30s Contact physical network operations colleagues to collect switch logs; troubleshoot hardware issues
RDMA communication retransmission exceeds the limit RDMA communication retransmission exceeds the limit Check network configuration and network bandwidth
BackendConfig configuration parameter validation failed MindIE configuration parameters are invalid Check configuration file parameters
Computing frequency reduction Abnormal decrease in NPU frequency Check cooling and power supply status
CPU preemption (partial process preemption) Some training processes experience CPU resource preemption Check process resource usage
Link congestion anomaly Some communication links experience conflict congestion Check switch routing policy

3.6 Diagnostic Result Output Directory

诊断结果输出目录/
└── fault_diag_result/
    ├── diag_report.json           # 诊断结果
    └── diag_report_{实例名}.json   # 多实例推理诊断结果

3.7 Notes

  1. Disk space: The disk space of the output directory specified by the cleansing command must be greater than 5G
  2. Sensitive information: During cleansing, the log files collected by the user will be read. Please confirm that there is no sensitive information in the directory.
  3. Single-machine cleansing: The directory to be cleansed contains only the original logs of a single training device
  4. Parameter selection：
    • Not specified --performance: Only root cause node analysis and fault event analysis are performed
    • Specify --performance: Perform all diagnostic modules (including device resource analysis and network congestion analysis)
  5. Diagnostic scenarios：
    • Root cause node analysis and fault event analysis appear → training task exits abnormally
    • No root cause node is diagnosed and fault event analysis has no result → performance degradation problem (will not cause abnormal training exit)
  6. Log level: When the log level configuration is low, there may be situations where log flooding flushes out key logs and diagnosis cannot be performed. Involved environment variable:ASCEND_GLOBAL_EVENT_ENABLE、HCCL_ENTRY_LOG_ENABLE、ASCEND_GLOBAL_LOG_LEVEL、ASCEND_MODULE_LOG_LEVEL
  7. MindIE multi-instance: Fault diagnosis of MindIE multi-instance inference clusters that have experienced (with/without) redundancy recovery is not currently supported

Reference documents