SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsexternal-gitcode-ascend-atb-debug-guide

external-gitcode-ascend-atb-debug-guide

Coding & development

View repository

Updated 10-15 17:19

About this skill

ATB Debug Guide

Function Overview

Provides an ATB debugging guide, covering environment configuration issues, ABI version mismatch, memory errors, CSV test failures, ACLNN interface issues, and other common scenarios.

When to Invoke

Invoke this skill in the following situations:

  • The user encounters ATB operator testing issues
  • The user needs to analyze the cause of an error
  • The user needs to understand ATB environment configuration

⚠️ Path Constraints (Must Be Followed)

Before executing this skill,the following paths must be obtained from the user(if not provided, use the default paths):

  • <CANN_PATH>: CANN installation path (e.g., /usr/local/Ascend/cann-9.0.0-beta.2）
  • <ATB_REPO_PATH>: ATB repository path

Index

Issue Type Quick Locate
ABI version mismatch undefined symbol, shared object file
Memory error ERROR_OUT_OF_HOST_MEMORY
Tensor dimension error ERROR_INVALID_TENSOR_DIM
Data type error KeyError: 'float32', ERROR_INVALID_TENSOR_INI_MATCH
ACLNN function signature error 161001, 161002, undefined symbol
ACLNN MoE scenario mismatch {xxx}Quant and other non-MoE operators
Log analysis methods plog logs, atb logs

Common Error Categories

ABI Version Mismatch Error

Error Message

OSError: /path/to/atb/output/atb/cxx_abi_0/lib/libatb_test_framework.so: cannot open shared object file

or

undefined symbol: _ZN3atb13OpParamToJson...

Cause

The ATB output directory has two ABI versions:

  • cxx_abi_0: Old version (g++/libstdc++abi_0)
  • cxx_abi_1: New version (g++/libstdc++abi_1)

The test framework library and dependency libraries must use the same ABI version.

Solution

Option A: Use the correct set_env.sh parameters

source <CANN_PATH>/set_env.sh
source <ATB_REPO_PATH>/output/atb/set_env.sh --cxx_abi=1
echo $ATB_HOME_PATH

Option B: Create a symbolic link (not recommended)

ln -s <ATB_REPO_PATH>/output/atb/cxx_abi_1 <ATB_REPO_PATH>/output/atb/cxx_abi_0

Verification Method

ls -la $ATB_HOME_PATH/lib/libatb_test_framework.so
ldd $ATB_HOME_PATH/lib/libatb_test_framework.so | grep libatb

Memory Error

Error Message

ActualError: S:ERROR_OUT_OF_HOST_MEMORY

Possible Causes

  1. Mismatch in the number of input/output tensors: The IN_TENSOR_NUM/OUT_TENSOR_NUM defined in the Runner do not match the interface specification
  2. Shape calculation error: Incorrect output shape calculation causes memory allocation failure
  3. Data too large: The shape of the test case exceeds the device memory limit

Solution

  1. Check whether IN_TENSOR_NUM and OUT_TENSOR_NUM in the Runner match the interface specification
  2. Check the shape inference logic in InferShapeImpl
  3. Reduce the shape size of the test case

Interface Specification Checkpoints

// 检查接口规格中的输入输出定义
// 例如 SwigluQuant:
// - 输入：1个 [ntokens, 2*hidden_size]
// - 输出：2个 [ntokens, hidden_size] + [ntokens]
// Runner 必须匹配这些规格

Tensor Dimension Error

Error Message

ActualError: I:ERROR_INVALID_TENSOR_DIM_NUM
ActualError: I:ERROR_INVALID_TENSOR_DIM

Checkpoints

  1. Whether the number of input tensor dimensions meets expectations (usually 2D)
  2. Whether specific dimension values are within the valid range
  3. Whether the dimension can be evenly divided by the specified value (e.g., hidden size must be divisible by 2)

Data Type Error

Error Message

ActualError: I:ERROR_INVALID_TENSOR_INI_MATCH
KeyError: 'float32'

Cause

  1. The input data type is not supported (e.g., float32 is not in the dtype_dict of data_generation.py)
  2. The output data type does not match expectations

Solution

Data types supported by data_generation.py in CSV:

  • float (corresponds to float32)
  • float16 / half
  • bf16 / bfloat16
  • int8 / char
  • int32 / int
  • int64 / long

ACLNN function signature mismatch

Error message

undefined symbol: aclnn{xxx}QuantGetWorkspaceSize
ActualError: S:ERROR_CANN_ERROR (161001, 161002)

Cause

  1. Function name spelled incorrectly
  2. Number/types of function parameters do not match the ACLNN API
  3. Dynamic library loading failed

Solution

# 检查函数符号
nm -D <ATB_PATH>/output/atb/cxx_abi_1/lib/libatb.so | grep -i {xxx}

nm -D $ASCEND_TOOLKIT_HOME/lib64/libopapi_nn.so | grep -i {xxx}

ACLNN error codes

Error code Name Meaning
161001 ACLNN_ERR_PARAM_NULLPTR Parameter is a null pointer (e.g., x/yOut is null)
161002 ACLNN_ERR_PARAM_INVALID Parameter type/dimension/value is invalid

ACLNN MoE scenario mismatch ⭐️

Problem description

Problem: The ATB operator is designed for non-MoE scenarios, but the ACLNN interface only supports MoE scenarios.

Typical case

{xxx}Quant:

  • ATB design: non-MoE Per-Token dynamic quantization
  • ACLNN: only supports MoE grouped dynamic quantization, requires groupIndex and smoothScales

Error message

[ERROR] errno[161001] Check yOut != nullptr failed

In fact, because smoothScalesOptional and groupIndexOptional do not support null pointers (in the current version).

Solution

Option A: Fall back to the OPS implementation (recommended)

// 禁用 ACLNN 路径
std::shared_ptr<Runner> XxxOperation::CreateRunner(Context &context) const
{
    // TODO: 待 ACLNN 支持非 MoE 场景后启用
    return std::make_shared<XxxOpsRunner>(param_);
}

Option B: Construct MoE parameters to simulate non-MoE (complex, not recommended)

  • Construct a single group groupIndex
  • Construct all-1 smoothScales

How to identify

Consult the ACLNN documentation:

"Currently xxx only supports MoE scenarios"

Log analysis methods

Log locations

Log type Path Content
ATB logs <MY_DIR>/log/<log_name>/atb/atb_*.log ATB operator scheduling, infershape, setup
Lower-layer component logs <MY_DIR>/log/<log_name>/debug/plog/plog-*.log GE、RUNTIME、OP、ASCENDCL

ATB log keywords

# 查找 ACLNN 加载
grep -i "aclnn\|LoadMethod\|workspace" atb_*.log

# 查找错误
grep -i "error\|fail\|ERROR" atb_*.log

# 查找测试用例执行
grep -i "Case\|times" atb_*.log

Lower-layer component log keywords

# 查找 ACLNN 底层错误
grep -i "161001\|161002\|errno" plog-*.log

# 查找 kernel 执行
grep -i "kernel\|execute" plog-*.log

# 查找内存分配
grep -i "malloc\|aclrtMalloc" plog-*.log

Typical log analysis workflow

  1. ATB layer (atb_*.log): check operator creation, Runner selection, shape inference
  2. Interface layer (plog-*.log OP/GE): check ACLNN interface calls, parameter validation
  3. Runtime layer (plog-*.log RUNTIME): check kernel execution, memory allocation

ATB environment check workflow

Step 1: Check NPU devices

npu-smi info

Step 2: Source the environment

source <CANN_PATH>/set_env.sh
source <ATB_REPO_PATH>/output/atb/set_env.sh
# 或指定 ABI 版本
source <ATB_REPO_PATH>/output/atb/set_env.sh --cxx_abi=1

Step 3: Verify the test framework

echo "ATB_HOME_PATH=$ATB_HOME_PATH"
ls -la $ATB_HOME_PATH/lib/libatb_test_framework.so

Step 4: Run the CSV test

cd <ATB_REPO_PATH>/tests/framework/python/CsvOpsTestTool
python3 atb_csv_ops_test.py -i <CSV_FILE_PATH>

Debugging checklist

Check item Command Expected result
NPU device npu-smi info Displays available devices
ATB_HOME_PATH echo $ATB_HOME_PATH Points to the cxx_abi_0/1 directory
Test framework library ls $ATB_HOME_PATH/lib/libatb_test_framework.so File exists
Dependency library versions ldd $ATB_HOME_PATH/lib/libatb_test_framework.so All dependencies resolvable
ACLNN symbol nm -D $ASCEND_TOOLKIT_HOME/lib64/libopapi_nn.so | grep aclnn Symbol exists

New issue record

## 问题名称

**错误信息：**

[paste error message]


**日志分析：**
- ATB 日志: [关键行]
- 下层日志: [关键行]

**根本原因：**
[分析原因]

**解决方案：**
[解决方案]

**验证方法：**
[验证命令]

Version: 1.1.0
Updated: 2026-04-25 - Format standardization