external-gitcode-ascend-fault_diagnose

Fault Diagnosis

Function Overview

The Ascend fault diagnosis tool is used to detect and analyze NPU-related faults, helping users quickly locate and resolve issues.

Prerequisites

Install ascend-fault-diagnose: Check whether it is installed using the ascend-fd version command
Installation Tutorial：https://www.hiascend.com/document/detail/zh/mindcluster/730/faultdiag/faultdiagug/mindxdlFDUG008.html

Usage Instructions

Usage Recommendations

Cluster Specification Limit: Due to the Linux system's maximum process count limit (default 1024), the cluster specification is recommended to be ≤128 servers (1024 cards). If the number of servers exceeds this specification, use the ulimit -n command to adjust the file descriptor limit
Avoid Pipe Commands: When using diagnostic tool commands, avoid using pipe commands as much as possible, as they may affect the acquisition of user IPs and log auditing

Supported Scenarios

Full-card training and inference on a whole machine: Fault diagnosis is supported only for full-card training and inference tasks on a whole machine. If the scenario is not full-card, fault root cause localization may be incorrect or fail
IPv4 Environment: Currently only IPv4 is supported; IPv6 is not supported

System Time Description

Synchronize Server Time: Please synchronize the system time of each training and inference server. Inconsistent system times may cause inaccurate analysis results
Synchronize Host and Device Time: Please synchronize the Host system time and the Device system time on each server
Synchronize Container Time: If containers are used to execute tasks, please synchronize the time between the host machine and the containers

Usage Restrictions

Log Version Matching Table

Log File	Corresponding Software	Software Version	Description
CANN application logs	CANN	7.0.RC1 and above	Host-side and Device-side application logs
PyTorch framework training and inference logs	PyTorch1.11.0 framework adaptation plugin	5.0.RC3 and above	-
MindSpore framework training logs	MindSpore	2.1.0 and above	Some fault types include version descriptions
TensorFlow framework training logs	TensorFlow	-	Only user-defined TensorFlow faults are supported
Host OS logs	-	-	Supports CentOS 7.6, Debian 10.0, EulerOS 2.10/2.12, CTyunOS 22.06, etc. The recommended log size is ≤512MB
Device-side logs	Ascend HDK	23.0.RC3 and above	-
MindCluster component logs	Device Plugin/NodeD, etc.	6.0.RC3 and above	-
MindIE component logs	MindIE Server/LLM, etc.	6.0.0 and above	-
AMCT component logs	AMCT	7.0.RC1 and above	Released integrated in the CANN package
MindIE Pod console logs	MindIE Pod	-	-

Diagnosis Process

日志采集 → 日志清洗 → 故障诊断

Information to be provided before starting diagnosis

Before starting fault diagnosis, the user needs to provide the following information:

Fault occurrence stage: The stage of training and inference (before/during/after)
- Before: Before training and inference start, collect NPU environment check files
- During: During training and inference, collect NPU status monitoring, host resource information, etc.
- After: After training and inference end, collect complete logs for diagnosis
Node connection data: Provide connection information for all nodes, used for log collection and cleaning
- Node IP address list
- SSH connection information (username, password, or key)
- Node role (master/worker)

Log collection script

Use the log_collector.py script to automatically collect logs:

Node configuration file format（JSON）：

[
    {"ip": "192.168.1.1", "user": "root", "password": "xxx", "name": "master"},
    {"ip": "192.168.1.2", "user": "root", "key_file": "/path/to/key", "name": "worker-1"}
]

Usage：

# 训练前采集
python3 log_collector.py -s before -n nodes.json -o ./logs

# 训练中采集
python3 log_collector.py -s during -n nodes.json -o ./logs

# 训练后采集
python3 log_collector.py -s after -n nodes.json -o ./logs

Parameter description：

Parameter	Description
`-s, --stage`	Collection stage: before/during/after
`-n, --nodes`	Node configuration file path
`-o, --output`	Output directory

I. Log collection

1.1 Log collection before training and inference

File：npu_info_before.txt
Path：采集目录/environment_check/

Collection command：

Tool	Command	Description
hccn_tool	`-net_health -g`	Network health status
hccn_tool	`-link -g`	RoCE link status
hccn_tool	`-ip -g`	IP and mask
hccn_tool	`-stat -g`	Sent and received packet statistics
hccn_tool	`-link_stat -g`	Historical link statistics
npu-smi	`info`	Basic device information
npu-smi	`info -t ecc`	ECC count
npu-smi	`info -t board`	Hardware information
npu-smi	`info -t usages`	Memory usage
npu-smi	`info -c 0 -t health`	Chip health status

1.2 Log collection during training and inference

File type	Naming constraint	Collection method
NPU network port statistics	`npu_{id}_details.csv`	hccn_tool -stat, every 15 seconds
NPU status monitoring	`npu_smi_{id}_details.csv`	npu-smi, every 15 seconds
Host resource information	`host_metrics_{core_num}.json`	top command
MindIE Pod logs	`{pod_name}.json`	kubectl logs

1.3 Log collection after training and inference

Log type	Naming constraint	Storage path
NPU environment check	`npu_info_after.txt`	`environment_check/`
User training logs	`rank-.log`、`worker-.log`	Collection directory
CANN application logs	`plog-.log`、`device-.log`	`process_log/`
Host-side logs	`messages`、`dmesg`	Collection directory
Device-side logs	`device-os_*.log`	`device_log/`
MindCluster logs	`devicePlugin*.log` etc.	`dl_log/`
MindIE logs	`mindie-*.log`	`mindie/log/debug/`
AMCT logs	`amct_*.log`	`amct_log/`

Host-side log collection command：

# 内核消息日志
dmesg -T | tail -n 100000 > 采集目录/dmesg

# 系统监测日志
cp /var/log/sysmonitor.log 采集目录/

# Device侧日志
msnpureport

MindCluster log collection command：

cp /var/log/mindx-dl/devicePlugin 采集目录/dl_log
cp /var/log/mindx-dl/noded 采集目录/dl_log
cp /var/log/ascend-docker-runtime 采集目录/dl_log
cp /var/log/mindx-dl/volcano-scheduler 采集目录/dl_log
cp /var/log/mindx-dl/volcano-controller 采集目录/dl_log
cp /var/log/mindx-dl/npu-exporter 采集目录/dl_log

1.4 Collection directory structure

采集目录/
├── messages                    # 主机侧操作系统日志
├── dmesg                       # 主机侧内核消息日志
├── sysmonitor.log              # 主机侧系统监测日志
├── rank-*.txt                  # 训练控制台日志
├── process_log/                # CANN应用日志
├── device_log/                 # Device侧日志
├── dl_log/                     # MindCluster组件日志
├── mindie/                     # MindIE组件日志
├── amct_log/                   # AMCT组件日志
└── environment_check/          # NPU环境检查文件
    ├── npu_info_before.txt
    ├── npu_info_after.txt
    ├── npu_smi_*_details.csv
    ├── npu_*_details.csv
    └── host_metrics_*.json

II. Log cleaning

2.1 Environment configuration

# 配置环境变量（普通用户）
export PATH=$PATH:/usr/local/python3.7.5/bin

# 验证配置
ascend-fd -h

2.2 Execute cleaning

# 创建输出目录
mkdir 清洗输出目录

# 执行清洗命令
ascend-fd parse -i 采集目录 -o 清洗输出目录 --performance

2.3 Cleaning output files

File	Description
`ascend-kg-parser.json`	Fault event analysis cleaning result (inference engine input)
`ascend-kg-analyzer.json`	Fault event analysis cleaning result
`ascend-rc-parser.json`	Root cause node analysis cleaning result
`device_ip_info.json`	Device IP information
`nic_clean.csv`	Network congestion cleaning result
`nad_clean.csv`	Computing frequency reduction cleaning result
`process_*.csv`	CPU resource preemption cleaning result
`plog-parser-*.log`	Root cause node analysis cleaned logs

2.4 Cleaning service flow logs

Clean service flow logs through the Python API interface.

Import interface：

from ascend_fd import parse_fault_type

Call example：

results, err_msg_list = parse_fault_type(input_log_list)

Input parameter input_log_list format：

[
    {
        "log_domain": {
            "server": "10.1.1.1",
            "device": ["0", "1"]
        },
        "log_items": [
            {
                "item_type": "MindIE",
                "log_lines": [
                    "[ERROR] xxx",
                    "[ERROR] yyy"
                ]
            }
        ]
    }
]

Input parameter description：

Field	Parameter type	Required	Description
log_domain	Dictionary	Yes	Log domain
server	String	Yes	Server address
device	List	Yes	Full card information of the fault occurrence
log_items	List	Yes	Log item
item_type	String	Yes	Log type
log_lines	List	Yes	Log line to be parsed

Output parameter results format：

[
    {
        "error_type": "AISW_MindIE_MS_HttpServer_01",
        "fault_domain": "Software",
        "attribute": {
            "key_info": "",
            "component": "MindIE",
            "module": "MS",
            "cause": "Httpserver通信超时",
            "description": "等待时间超过设定的时延。",
            "suggestion": ["1. 请联系华为工程师处理；"]
        },
        "device_list": [
            {
                "server": "172.0.0.1",
                "device": ["0", "1", "2"]
            }
        ]
    }
]

Output parameter description：

Field	Parameter type	Description
error_type	String	Fault code
fault_domain	String	Fault domain
attribute	Dictionary	Fault attribute
key_info	String	Key log
component	String	Fault component
module	String	Fault module
cause	String	Fault cause
description	String	Fault description
suggestion	List	Recommended solution
device_list	List	List of devices where the fault occurred

Error information err_msg_list: List of error information generated during interface execution.

2.5 Root cause node cleaning and diagnosis

Clean and diagnose root cause nodes through the Python API interface.

Import interface：

from ascend_fd import parse_root_cluster
from ascend_fd import diag_root_cluster

Call example：

# 根因节点清洗
rc_parse_results, rc_parse_err_msg = parse_root_cluster(input_log_list)

# 根因节点诊断
results, err_msg_list = diag_root_cluster(rc_parse_results)

Input parameter input_log_list format：

[
    {
        "log_domain": {
            "server": "10.1.1.1",
            "instance_id": "instance_name"
        },
        "log_items": [
            {
                "item_type": "plog",
                "pid": 3199,
                "device_id": 0,
                "rank_id": 0,
                "log_lines": ["[ERROR] xxx."]
            }
        ]
    }
]

Input parameter description：

Field	Parameter type	Required	Description
log_domain	Dictionary	Yes	Log domain
server	String	Yes	Server address
instance_id	String	Yes	Instance ID
log_items	List	Yes	Log item
item_type	String	Yes	Log type
pid	Int	Yes	Process ID
device_id	Int	Yes	Device ID
rank_id	Int	Yes	Rank ID
log_lines	List	Yes	Log line to be parsed

Output parameter results format：

{
    'analyze_success': True,
    'fault_description': {
        'code': 102,
        'string': '所有有效节点的Plog都没有错误日志信息，无法定位根因节点。'
    },
    'root_cause_device': ['ALL Device'],
    'device_link': [],
    'remote_link': '',
    'first_error_device': '',
    'last_error_device': ''
}

Output parameter description：

Field	Parameter type	Description
analyze_success	Bool	Whether the diagnosis is successful (True/False)
fault_description	Dictionary	Fault description
code	Int	Fault code
string	String	Fault code description
root_cause_device	List	Root cause device information
device_link	List	Root cause node chain
remote_link	String	Inter-card wait chain
first_error_device	String	Device with the earliest error
last_error_device	String	Device with the latest error

Error information：

rc_parse_err_msg: List of error information generated during cleaning
err_msg_list: List of error information generated during diagnosis

2.6 Fault event cleaning and diagnosis

Clean and diagnose fault events through the Python API interface.

Import Interface：

from ascend_fd import parse_knowledge_graph
from ascend_fd import diag_knowledge_graph

Call Example：

# 故障事件清洗
kg_parse_results, kg_parse_err_msg = parse_knowledge_graph(input_log_list, custom_entity)

# 故障事件诊断
results, err_msg_list = diag_knowledge_graph(kg_parse_results)

Input Parameter input_log_list Format：

[
    {
        "log_domain": {
            "server": "10.1.1.1"
        },
        "log_items": [
            {
                "item_type": "MindIE",
                "path": "/log/debug/mindie-ms_11_202411061400.log",
                "device_id": 0,
                "modification_time": "2025-08-21 23:50:59.999999",
                "component": "Controller",
                "log_lines": ["[ERROR] xxx."]
            }
        ]
    }
]

Input Parameter Description：

Field	Parameter Type	Required	Description
log_domain	Dictionary	Yes	Log domain
server	String	Yes	Server address
log_items	List	Yes	Log item
item_type	String	Yes	Log type
path	String	No	Log file path (required when cleaning NPU environment check files)
device_id	Int	No	Device card number
modification_time	String	No	Log modification time (as the fault occurrence time)
component	String	No	Component name (supports Coordinator and Controller)
log_lines	List	Yes	Log line to be parsed

Custom Fault Entity custom_entity Format：

{
    "41001": {  # 故障码（自定义，不能与已有故障码相同）
        "attribute.class": "Software",
        "attribute.component": "AI Framework",
        "attribute.module": "Compiler",
        "attribute.cause_zh": "抽象类型合并失败",
        "attribute.description_zh": "对函数输出求梯度时，抽象类型不匹配，导致抽象类型合并失败。",
        "attribute.suggestion_zh": [
            "1. 检查求梯度的函数的输出类型与sens_param的类型是否相同，如果不相同，修改为相同类型；",
            "2. 自动求导报错Type Join Failed"
        ],
        "attribute.error_case": [
            "grad = ops.GradOperation(sens_param=True)",
            "# test_net输出类型为tuple(Tensor, Tensor)",
            "def test_net(a, b):",
            "    return a, b"
        ],
        "attribute.fixed_case": [
            "grad = ops.GradOperation(sens_param=True)",
            "# test_net输出类型为tuple(Tensor, Tensor)",
            "def test_net(a, b):",
            "    return a, b"
        ],
        "rule": [{"dst_code": "20106"}],
        "source_file": "TrainLog",
        "regex.in": ["Abstract type", "cannot join with"]
    }
}

custom_entity Parameter Description：

Field	Parameter Type	Required	Description
attribute.class	String	Yes	Fault category
attribute.component	String	Yes	Fault component
attribute.module	String	Yes	Fault module
attribute.cause_zh	String	Yes	Fault cause
attribute.description_zh	String	Yes	Fault description
attribute.suggestion_zh	List	Yes	Suggested solution list
attribute.error_case	List	No	Error example code
attribute.fixed_case	List	No	Fix example code
rule	List	Yes	Fault propagation rule
source_file	String	Yes	Log source file type
regex.in	List	Yes	Match regular expression list

Output Parameter results Format：

[
    {
        'analyze_success': True,
        'version_info': {},
        'note': '',
        'fault': [{
            'code': 'NORMAL_OR_UNSUPPORTED',
            'component': '',
            'module': '',
            'cause_zh': '故障事件分析模块无结果',
            'description_zh': '可能为正常训练作业，无故障发生。',
            'suggestion_zh': '1. 若存在问题无法解决，请联系华为工程师处理',
            'class': '',
            'fault_source': ['1.1.1.1 device-Unknown'],
            'fault_chains': []
        }]
    }
]

Output Parameter Description：

Field	Parameter Type	Description
analyze_success	Bool	Whether diagnosis is successful (True/False)
version_info	Dictionary	Version information
note	String	Remarks
fault	List	Fault event list
code	String	Fault code
component	String	Fault component
module	String	Fault module
cause_zh	String	Fault cause
description_zh	String	Fault description
suggestion_zh	String	Suggested solution
class	String	Fault category
fault_source	List	Fault source
fault_chains	List	Fault propagation chain

Error information：

kg_parse_err_msg: List of error information generated during cleaning
err_msg_list: List of error information generated during diagnosis

III. Fault Diagnosis

3.1 Cluster Fault Diagnosis

# 创建诊断结果输出目录
mkdir 诊断结果输出目录

# 执行诊断命令（默认仅诊断根因节点、故障事件两个模块）
ascend-fd diag -i 诊断输入目录 -o 诊断结果输出目录

# 执行完整诊断（包含性能劣化检测模块）
ascend-fd diag -i 诊断输入目录 -o 诊断结果输出目录 --performance

3.2 Standalone Fault Diagnosis

Standalone diagnosis returns the corresponding data of the fault event module by default, which is suitable for single-node fault troubleshooting.

# 创建单机诊断结果输出目录
mkdir 单机诊断结果输出目录

# 执行单机诊断命令
ascend-fd single-diag -i 采集目录 -o 单机诊断结果输出目录

Standalone diagnosis output directory：

单机诊断结果输出目录/
└── fault_diag_result/
    └── diag_report.json    # 诊断结果

Standalone diagnosis description：

Standalone diagnosis scans fault events of all valid logs in the node
If fault event analysis appears in the echo, it indicates that the current fault may cause abnormal exit of training or inference tasks
When an execution error occurs, you can use the diag_report.json file to view all exception information

3.3 Super Node Fault Diagnosis

Super node fault diagnosis provides three scenarios:

Super node topology information non-manual association scenario
Super node topology information manual association scenario
Missing Host log scenario

3.3.1 Non-Manual Association Scenario

Requires that all three types of logs—BMC, Host, and LCNE—exist simultaneously; none of the log types may be missing.

# 创建超节点诊断结果输出目录
mkdir 超节点诊断结果输出目录

# 执行超节点诊断命令
ascend-fd diag -i 诊断输入目录 -o 诊断结果输出目录 -s super_pod

Cleaning result directory structure：

超节点清洗结果输出目录/
├── bmc/
│   ├── bmc_xxx.xx.xx.xx4_1/
│   │   ├── ascend-kg-analyzer.json
│   │   ├── ascend-kg-parser.json
│   │   └── server-info.json
│   └── bmc_xxx.xx.xx.xx5_1/
│       └── ...
├── host/
│   ├── log_collect_node-29-121_20250616/
│   │   ├── ascend-kg-analyzer.json
│   │   ├── ascend-kg-parser.json
│   │   ├── ascend-rc-parser.json
│   │   └── server-info.json
│   └── log_collect_node-29-124_20250616/
│       └── ...
└── lcne/
    ├── xxx.xx.xx.xx6/
    │   └── ...
    └── xxx.xx.xx.xx7/
        └── ...

Diagnosis result output：

fault_diag_result/
├── diag_report.json    # 诊断结果
└── topo_info.json      # 超节点拓扑信息

3.3.2 Manual Association Scenario

During cleaning, the three types of logs—BMC, Host, and LCNE—need to be manually associated.

Cleaning command example：

ascend-fd parse \
  --host_log parse_input/host/xxx.xx.xx.131/host_log/ \
  --mindie_log parse_input/host/xxx.xx.xx.131/mindie/ \
  --process_log parse_input/host/xxx.xx.xx.131/process_log/ \
  --bmc_log parse_input/bmc/worker-104 \
  --lcne_log parse_input/lcne/worker-204 \
  -o 清洗结果输出目录/worker-1