ascendc-mssanitizer

Ascend C operator mssanitizer memory detection and analysis

Systematically detect memory issues in Ascend C operators and generate detailed analysis reports.

Overview

mssanitizer (MindStudio Sanitizer) is a memory correctness detection tool suite provided by CANN, used to detect memory issues in AscendC operator development, including:

Memory Leak
Illegal Access
Uninitialized Memory
Data Race
Sync Issues

Trigger Conditions

The user needs to perform memory detection on an Ascend C operator
The user requests memory leak detection
The user needs to verify operator memory safety
The user mentions keywords such as "mssanitizer", "memory detection", "memory leak", "illegal access", etc.

Project Type Determination and Detection Mode Selection

Key Step: Before performing detection, the operator project type must first be determined, and the corresponding detection mode selected.

Determination Logic

算子工程目录
├── 存在 op_graph/ 目录？ ─── 是 ──→ ops 算子仓 → C++ 模式
├── 存在 op_host/op_api/ 目录？ ── 是 ──→ ops 算子仓 → C++ 模式
├── 存在 examples/test_geir_*.cpp？ ── 是 ──→ ops 算子仓 → C++ 模式 (GE IR 子模式)
├── 存在 examples/test_aclnn_*.cpp？ ── 是 ──→ ops 算子仓 → C++ 模式 (aclnn 子模式)
└── 以上均无 ──→ 自定义算子仓 → Python 模式

Detection Mode Comparison

Feature	Python Mode	C++ Mode (GE IR)	C++ Mode (aclnn)
Applicable Project	Custom operator repository	ops operator repository	ops operator repository
Project Characteristics	No op_graph/, no examples/	Has op_graph/, has test_geir_*.cpp	Has examples/, has test_aclnn_*.cpp
Test Carrier	Python script (torch_npu invocation)	C++ executable (GE IR graph invocation)	C++ executable (aclnn API invocation)
Test Script	`gen_test_script.py` generated	`examples/test_geir_*.cpp`	`examples/test_aclnn_*.cpp`
Execution Script	`run_mssanitizer.sh`	`run_mssanitizer_geir.sh`	`run_mssanitizer_geir.sh`
Link Library	torch_npu	ascendcl, ge_runner, graph, register	ascendcl, opapi, nnopbase
Log Prefix	`memcheck_device_`	`geir_memcheck_device_`	`geir_memcheck_device_`
Summary Report	`mssanitizer_summary_`	`geir_mssanitizer_summary_`	`geir_mssanitizer_summary_`
Advantages	Covers multiple shapes/dtypes	Directly invokes the operator kernel, closer to real execution	No op_graph required, simpler to build
Disadvantages	Fallback may bypass the custom operator	Requires compiling a C++ executable	Requires compiling a C++ executable

Note：run_mssanitizer_geir.sh The script already automatically supports both C++ sub-modes, and will preferentially search for test_geir_*.cpp, and automatically fall back to test_aclnn_*.cpp。

Tool Scripts

This skill provides the following scripts, located in the scripts/ directory:

Script	Purpose	Mode
`scripts/gen_test_script.py`	Automatically generate a Python test script based on the operator name	Python
`scripts/run_mssanitizer.sh`	Python mode: execute all 5 detection items and generate a summary report	Python
`scripts/run_mssanitizer_geir.sh`	C++ mode: automatically build + detect + generate a summary report (supports GE IR and aclnn)	C++
`scripts/parse_mssanitizer_log.py`	Parse memcheck logs to generate problem analysis reports	General

SKILL root directory: /home/rcz/agent-skills/skills/ascendc-operator-mssanitizer

Prerequisites

Required environment variables

Environment variable	Description	How to obtain
`ASCEND_HOME_PATH`	CANN installation path	Set via CANN set_env.sh
`LD_LIBRARY_PATH`	Dynamic library path	Includes CANN lib64

Environment activation

source /root/miniconda3/bin/activate cann_env
source /root/miniconda3/envs/cann_env/Ascend/cann-8.5.0/set_env.sh

Key constraints (practical experience)

CANN version locking: Multiple CANN installations may exist in the container (such as cann-8.5 and cann-9.0),set_env.sh set ASCEND_HOME_PATH may point to the wrong version. The script explicitly overrides it with the user-specified path to ensure mssanitizer uses the correct version.
--check-device-heap and --check-cann-heap are mutually exclusive: They cannot be enabled at the same time; memcheck must be run twice separately.
libhccl.so dependency: After mssanitizer modifies LD_LIBRARY_PATH , libhccl.somay be lost, causing torch_npu import to fail. The script has automatically completed the CANN lib64 path.
Container compatibility: The script only uses grep(not rg), ensuring it is available inside Docker containers.
Empty log = no errors: mssanitizer writes to the log file only when errors are detected. A log file size of 0 means the check item passed, not that the tool did not run.

Detection tool types

mssanitizer provides four detection tools:

Tool	Command parameter	Detection content
Memory detection	`-t memcheck`	Illegal memory access, illegal free, memory leaks, UB out-of-bounds access
Race detection	`-t racecheck`	Multi-core parallel race conditions, data races
Uninitialized detection	`-t initcheck`	Uninitialized memory reads
Synchronization detection	`-t synccheck`	Synchronization errors, synchronization point issues

Common parameter description

Parameter	Description	Default value
`-t <tool>`	Specify the detection tool	memcheck
`--leak-check=<yes/no>`	Whether to detect memory leaks	no
`--check-device-heap=<yes/no>`	Detect Device interface leaks	no
`--check-cann-heap=<yes/no>`	Detect AscendCL interface leaks (mutually exclusive with device-heap mutually exclusive）	no
`--log-file=<file>`	Log output file	stdout
`--log-level=<level>`	Log level (warn)	warn
`--kernel-name=<name>`	Only detect kernels with the specified name	all
`--block-id=<id>`	Only detect the specified block	all blocks
`--cache-size=<size>`	Single-block record cache size (MB)	100

Detection workflow

━━━━━━━━━━ Mode A: Python mode (custom operator repository) ━━━━━━━━━━

Suitable for custom operator projects without op_graph/ or examples/.

Stage 1: Generate test script

Use gen_test_script.py to automatically generate a test script for the specified operator:

SKILL_DIR="/home/rcz/agent-skills/skills/ascendc-operator-mssanitizer"

python3 "${SKILL_DIR}/scripts/gen_test_script.py" \
    --operator <operator_name> \
    --fallback <fallback_function> \
    --dtypes float16 float32 \
    --output <project>/mssanitizer_test/<operator_name>_mssanitizer_test.py

Parameter description：

--operator: Custom operator name (corresponds to the fallback function name in torch.ops.customize.<name>）
--fallback：torch.nn.functional (such as gelu、relu). Used automatically when the custom operator is not registered, ensuring the NPU path is still executed and monitored by mssanitizer
--dtypes: List of data types to test (only use types actually supported by the operator)

Stage 2: Run Detection

bash "${SKILL_DIR}/scripts/run_mssanitizer.sh" \
    <project>/mssanitizer_test/<operator_name>_mssanitizer_test.py \
    <cann_root>

Parameter Description：

1st parameter: Test script path (generated in Stage 1)
2nd parameter: CANN installation root path (optional, defaults to $ASCEND_HOME_PATH）

This script automatically executes:

memcheck (device-heap) — --check-device-heap=yes --leak-check=yes
memcheck (cann-heap) — --check-cann-heap=yes --leak-check=yes
racecheck — Data race detection
initcheck — Uninitialized memory detection
synccheck — Synchronization issue detection
Generate summary report — mssanitizer_summary_<timestamp>.md
Parse memcheck logs — Call parse_mssanitizer_log.py to generate detailed analysis report

Stage 3: Parse Logs Separately (Optional)

python3 "${SKILL_DIR}/scripts/parse_mssanitizer_log.py" \
    ./mssanitizer_logs/memcheck_device_<timestamp>.log \
    --output ./mssanitizer_logs/memcheck_analysis_report.md

━━━━━━━━━━ Mode B: C++ Mode (ops operator repository) ━━━━━━━━━━

Applicable to ops operator repositories (such as the ops-nn repository), where the project directory contains directories such as op_graph/、examples/、op_host/op_api/ .

Features

Invoke operators through C++ executables, closer to the real execution path
Automatically identify test source file type:
- GE IR sub-mode: use examples/test_geir_*.cpp, link ascendcl + ge_runner + graph + register
- aclnn sub-mode: use examples/test_aclnn_*.cpp, link ascendcl + opapi + nnopbase
Automatically build C++ executable (if not yet built)
Log files are distinguished by the geir_ prefix

One-Click Execution

SKILL_DIR="/home/rcz/agent-skills/skills/ascendc-operator-mssanitizer"

bash "${SKILL_DIR}/scripts/run_mssanitizer_geir.sh" \
    <project_dir> \
    <cann_root>

Parameter Description：

1st parameter: Operator project directory path (e.g., /home/rcz/ops-nn/activation/gelu_quant）
2nd parameter: CANN installation root path (optional, defaults to $ASCEND_HOME_PATH）

The script automatically executes：

Detect operator project structure, locate examples/test_geir_*.cpp or examples/test_aclnn_*.cpp
Automatically build test executable (if not yet built), selecting GE IR or aclnn build method based on source file type
Sequentially execute 5 detection items (memcheck_device → memcheck_cann → racecheck → initcheck → synccheck)
Generate summary report geir_mssanitizer_summary_<timestamp>.md
Call parse_mssanitizer_log.py to generate detailed analysis report

Manual Step-by-Step Execution (Optional)

For finer-grained control, you can execute manually:

# 1. 构建测试可执行文件（首次需要）
cd <project>/mssanitizer_test/build
cmake . && make -j$(nproc)

# 2. 逐项执行检测
MSSAN=<cann_root>/tools/mssanitizer/bin/mssanitizer
LOGDIR=<project>/mssanitizer_logs
TS=$(date +%Y%m%d_%H%M%S)

$MSSAN -t memcheck --check-device-heap=yes --leak-check=yes \
    --log-file=$LOGDIR/geir_memcheck_device_$TS.log \
    -- ./test_geir_<op_name> float

$MSSAN -t memcheck --check-cann-heap=yes --leak-check=yes \
    --log-file=$LOGDIR/geir_memcheck_cann_$TS.log \
    -- ./test_geir_<op_name> float

$MSSAN -t racecheck --log-file=$LOGDIR/geir_racecheck_$TS.log \
    -- ./test_geir_<op_name> float

$MSSAN -t initcheck --log-file=$LOGDIR/geir_initcheck_$TS.log \
    -- ./test_geir_<op_name> float

$MSSAN -t synccheck --log-file=$LOGDIR/geir_synccheck_$TS.log \
    -- ./test_geir_<op_name> float

# 3. 解析日志
python3 ${SKILL_DIR}/scripts/parse_mssanitizer_log.py \
    $LOGDIR/geir_memcheck_device_$TS.log \
    --output $LOGDIR/geir_memcheck_device_analysis_$TS.md

Output Files

Python Mode Output

After detection completes, generated under <project>/mssanitizer_logs/ :

<project>/mssanitizer_logs/
├── memcheck_device_<ts>.log           # memcheck device-heap 原始日志
├── memcheck_device_report_<ts>.json   # memcheck device-heap 测试结果
├── memcheck_device_analysis_<ts>.md   # memcheck device-heap 解析报告
├── memcheck_cann_<ts>.log             # memcheck cann-heap 原始日志
├── memcheck_cann_report_<ts>.json
├── memcheck_cann_analysis_<ts>.md
├── racecheck_<ts>.log
├── racecheck_report_<ts>.json
├── initcheck_<ts>.log
├── initcheck_report_<ts>.json
├── synccheck_<ts>.log
├── synccheck_report_<ts>.json
└── mssanitizer_summary_<ts>.md        # 汇总报告

GE IR Mode Output

After detection completes, generated under <project>/mssanitizer_logs/ :

<project>/mssanitizer_logs/
├── geir_memcheck_device_<ts>.log      # memcheck device-heap 原始日志
├── geir_memcheck_cann_<ts>.log        # memcheck cann-heap 原始日志
├── geir_racecheck_<ts>.log
├── geir_initcheck_<ts>.log
├── geir_synccheck_<ts>.log
└── geir_mssanitizer_summary_<ts>.md   # 汇总报告（含解析）

Note: A log file size of 0 indicates that the detection item passed and no errors were detected. mssanitizer only writes log content when errors are detected.

Staged Detection Strategy

Based on the CANN software stack structure, memcheck locates memory leaks in two stages:

┌─────────────────────────────────────┐
│         用户代码（Host 侧）          │
├─────────────────────────────────────┤
│      AscendCL 接口（CANN API）       │
├─────────────────────────────────────┤
│     Device 接口（驱动层接口）         │
└─────────────────────────────────────┘

Step 1: Detect Device Interface Leaks

mssanitizer -t memcheck --check-device-heap=yes --leak-check=yes -- python3 test.py
# 或 GE IR 模式：
mssanitizer -t memcheck --check-device-heap=yes --leak-check=yes -- ./test_geir_op float

No leak output → Leak occurs in the Device-side application
Leak output present → Continue to Step 2

Step 2: Detect AscendCL Interface Leaks

mssanitizer -t memcheck --check-cann-heap=yes --leak-check=yes -- python3 test.py
# 或 GE IR 模式：
mssanitizer -t memcheck --check-cann-heap=yes --leak-check=yes -- ./test_geir_op float

Error Types and Severity Levels

Severity Classification

Severity	Error Type	Handling Priority
🔴 Severe	illegal_write, illegal_read, illegal_free	Fix immediately
🟡 Moderate	memory_leak	Fix as soon as possible
🟢 Minor	Uninitialized memory use	Recommended to fix

Log Key Field Description

Field	Description	Example
`at 0x... on GM`	Global memory address	`0x12c000000010`
`in block on device N`	Device and block information	`device 0`
`serialNo:N`	Instruction serial number, used to locate the code position	`serialNo:22`
`size N`	Number of bytes accessed	`size 8`

Problem Localization Tips

Use serialNo to locate: serialNo corresponds to the compiled instruction serial number; after adding the -g compilation option, the source code location can be obtained
Address analysis: The GM address can be used to analyze memory layout and out-of-bounds situations
Size analysis: The access size can help determine the data type and operation type
illegal_free correlation analysis: If illegal_free appears together with illegal_write, it is usually an out-of-bounds write that has corrupted the memory management structure; illegal_write should be fixed first

Common Problem Troubleshooting Checklist

1. DataCopy parameter error

Symptom: illegal write/read error

Troubleshooting steps：

Check DataCopy whether the third parameter is correct
Confirm whether the element count calculation is correct (note: the parameter is the number of elements, not the number of bytes)
Check whether the source/destination address is out of bounds

Typical error：

// 错误：乘以 sizeof(T) 或其他系数
DataCopy(xLocal, xGlobal, totalLength * sizeof(T));  // ❌
DataCopy(xLocal, xGlobal, totalLength * 2);           // ❌

// 正确：直接使用元素数量
DataCopy(xLocal, xGlobal, totalLength);               // ✓

2. Memory allocation/deallocation mismatch

Symptom: memory leak error

Troubleshooting steps：

Check all AllocTensor whether there is a corresponding FreeTensor
Check EnQue/DeQue whether they are used in pairs
Check pipe.InitBuffer buffer management

Typical error：

// 错误：忘记释放
LocalTensor<T> xLocal = xQueue.AllocTensor<T>();
// ... 使用 xLocal
// 缺少 xQueue.FreeTensor(xLocal);  // ❌

// 正确：配对使用
LocalTensor<T> xLocal = xQueue.AllocTensor<T>();
// ... 使用 xLocal
xQueue.FreeTensor(xLocal);  // ✓

3. Insufficient UB buffer size

Symptom: UB address out of bounds error

Troubleshooting steps：

Check pipe.InitBuffer whether the allocated size is sufficient
Confirm whether the data type size calculation is correct
Check whether 32-byte alignment is required

Typical error：

// 错误：缓冲区大小不足
pipe.InitBuffer(tmpBuffer, totalSize);  // ❌ 未考虑临时空间需求

// 正确：根据算子需求分配足够空间
pipe.InitBuffer(tmpBuffer, totalSize * sizeof(T) + EXTRA_SPACE);  // ✓

Troubleshooting

Problem: mssanitizer not found

Check items:

Ensure ASCEND_HOME_PATH points to the correct CANN root path
Ensure the CANN version supports mssanitizer (8.0+)
There may be multiple CANN installations in the container; the cann_root parameter must be passed explicitly

Problem: torch_npu import failed (libhccl.so)

Cause: mssanitizer modifies LD_LIBRARY_PATHwhen starting, causing the CANN lib64 path to be lost

Solution: run_mssanitizer.sh has automatically added the CANN lib64 path at the beginning of LD_LIBRARY_PATH . If it still fails, run manually:

export LD_LIBRARY_PATH=<cann_root>/lib64:$LD_LIBRARY_PATH

Problem: CANNOT enable both --check-device-heap and --check-cann-heap

Cause: These two options are mutually exclusive

Solution: Run memcheck in two separate runs (run_mssanitizer.sh / run_mssanitizer_geir.sh already handles this automatically)

Problem: Log shows `<unknown>:0`

Cause: The -g debug option was not added during operator compilation

Solution: Recompile the operator with the -g -O0 option added

Problem: Log file size is 0

Cause: mssanitizer writes to the log file only when an error is detected

Solution: A log of 0 bytes indicates that this check item passed and no action is needed. You can use the mssanitizer internal log (mindstudio_sanitizer_log/) to confirm whether the tool is running properly.

Issue: C++ mode build failed

Check item:

Confirm that examples/ exists in the directory test_geir_*.cpp or test_aclnn_*.cpp
GE IR mode: Confirm that the op_graph/ directory exists and contains the operator proto header files
aclnn mode: Confirm that CANN's lib64/ and aarch64-linux/lib64/ contain libopapi.so and libnnopbase.so
Confirm that the CANN environment is correctly activated (ASCEND_HOME_PATH、LD_LIBRARY_PATH）
Check for compilation errors in the CMake output
If linking libopapi.so fails, check the CANN architecture subdirectory path (aarch64-linux/lib64）

Issue: No error detected but the program fails

Check item:

Try other mssanitizer tools (racecheck, initcheck)
Check for logic errors
Use other debugging tools

Notes

Performance impact: mssanitizer significantly slows down program execution (possibly 10-100 times slower), so it is only used for debugging
Memory overhead: The detection process requires additional memory to record allocation information, which may require more memory
Multiple runs: Some issues may not reproduce every time, so it is recommended to run the detection multiple times
Mode selection: The ops operator repository prioritizes using C++ mode (the script automatically identifies GE IR / aclnn), which can directly call the operator kernel and produce more accurate detection results

Best practices

Add debug information: Add -g -O0 during compilation to obtain precise code locations
Run all detection modes: Run all detection modes (the script already runs all 5 items by default)
Fix by priority: Resolve illegal access issues first, then handle memory leaks
Re-verify: After fixing, run mssanitizer again to confirm that the issue has been resolved
Regular detection: Perform detection after operator development is completed, after modifying memory-related code, and before submitting code
Choose the correct mode: Use C++ mode for the ops operator repository (the script automatically identifies GE IR / aclnn), and use Python mode for custom operators

About this skill

Overview

Trigger Conditions

Project Type Determination and Detection Mode Selection

Determination Logic

Detection Mode Comparison

Tool Scripts

Prerequisites

Required environment variables

Environment activation

Key constraints (practical experience)

Detection tool types

Common parameter description

Detection workflow

━━━━━━━━━━ Mode A: Python mode (custom operator repository) ━━━━━━━━━━

Stage 1: Generate test script

Stage 2: Run Detection

Stage 3: Parse Logs Separately (Optional)

━━━━━━━━━━ Mode B: C++ Mode (ops operator repository) ━━━━━━━━━━

Features

One-Click Execution

Manual Step-by-Step Execution (Optional)

Output Files

Python Mode Output

GE IR Mode Output

Staged Detection Strategy

Error Types and Severity Levels

Severity Classification

Log Key Field Description

Problem Localization Tips

Common Problem Troubleshooting Checklist

1. DataCopy parameter error

2. Memory allocation/deallocation mismatch

3. Insufficient UB buffer size

Troubleshooting

Problem: mssanitizer not found

Problem: torch_npu import failed (libhccl.so)

Problem: CANNOT enable both --check-device-heap and --check-cann-heap

Problem: Log shows <unknown>:0

Problem: Log file size is 0

Issue: C++ mode build failed

Issue: No error detected but the program fails

Notes

Best practices

Related documents

Problem: Log shows `<unknown>:0`