external-gitcode-ascend-atb-csv-tester

ATB CSV Tester

Function Overview

Run ATB CSV test cases to verify operator correctness. It must be used with the CANN environment and a compiled ATB test framework.

Path Constraints

The following paths must be obtained from the user：

<CANN_PATH>: CANN installation path (e.g., /usr/local/Ascend/ascend-toolkit/latest）
<ATB_REPO_PATH>: ATB repository path

If a path is invalid, stop immediately and report an error.

When to Invoke

Run ATB CSV tests (e.g., linear.csv）
Verify the correctness of migrated operators
Operator-level functional testing in the CANN environment

Device Support Description

The current test environment only supports Ascend910B devices.

Execution Rules

When executing testsOnly run cases whose SocVersion includes Ascend910B(the test framework automatically matches the current device SocVersion)
When generating CSV test cases, they have already been separated based on device differencesseparate design(see atb-csv-testcase-generator multiple-device test design principles)

910B/950 Difference Case Separation Requirements

⚠️ When 910B and 950 differ in any of the following aspects, they must be split into independent cases, each specifying its own SocVersion:

Difference Dimension	Separation Requirement
dtype support	Design positive and negative cases separately for each supported dtype
format support	Test each legal format separately
parameter constraints	Verify each legal/illegal parameter value separately
dimension limits	Design each boundary value/UB upper limit separately
ExpectedError	The same input may trigger different error codes; design negative cases separately
performance benchmarks	Mark performance tests with their respective SocVersion separately

It is strictly forbidden to merge cases with functional differences into Ascend910B;Ascend950— merging SocVersion is only allowed when the two devices are completely identical.

Prompt After 910B Testing Is Complete

After all Ascend910B tests pass, the user must be prompted:

"910B testing has all been completed. The CSV test cases already include independent cases for Ascend950 (SocVersion does not include 910B). Please manually run the CSV tests on an Ascend950 device to verify 950 compatibility:"
cd <ATB_REPO_PATH>/tests/framework/python/CsvOpsTestTool
python3 atb_csv_ops_test.py -i <CSV_FILE_PATH>

The test command for the 950 device is the same; the user only needs to switch the test environment to an Ascend950 machine and re-execute
If there are failures in the 950 tests, compare with the 910B results to analyze whether it is expected behavior caused by device differences

Gate 5: Full Regression Gate

Mandatory rule: Before entering Phase 6 delivery, all CSV files related to the target operator must all be run and pass.

Related CSV Enumeration

Based on the target operator name (e.g., layer_norm), enumerate all related CSV files:

find <ATB_REPO_PATH>/tests/apitest/opstest/csv/ -name '{op_name}*.csv' -type f | sort

Matching Rule	Description	Example (layer_norm)
Exact match `{op_name}.csv`	Main operator CSV	`layer_norm.csv`
Prefix match `{op_name}_*.csv`	Variant/extension CSV	`layer_norm_test.csv`

Gate check process

[Phase 5b 开始]
    │
    ▼
[步骤 1: 枚举相关 CSV] → find csv/ -name '{op}*.csv'
    │
    ▼
[步骤 2: 逐个运行] → python3 atb_csv_ops_test.py -i <每个 CSV>
    │
    ▼
[步骤 3: 汇总结果] → 记录各 CSV 通过/失败/跳过
    │
    ▼
[Gate 5 检查]
    ├─ 全部通过 → Gate 5 通过 → 可进入 Phase 6
    └─ 任一失败 → Gate 5 阻断 → 返回修复

Script for when the gate blocks

## Gate 5 阻断 — 全量回归未通过

| CSV 文件 | 总用例 | 通过 | 失败 | 状态 |
|---------|--------|------|------|------|
| {op}_test.csv | 31 | 31 | 0 | ✅ |
| {op}.csv | 90 | - | - | 未执行 |

**在 Gate 5 通过前，禁止进入 Phase 6 交付。**

下一步：运行未执行的 CSV 文件，修复失败用例，重新执行全量回归。

After Phase 5a passes: automatically prompt to enter full regression

When all Phase 5a design test cases pass, the Agent must automatically output the following prompt:

## Phase 5a 全部通过 — 请进入 Phase 5b 全量回归

Phase 5a 设计用例已全部通过（N/N）。

### Phase 5b 需执行的全量回归 CSV 列表：

| # | CSV 文件 | 路径 | 
|---|---------|------|
| 1 | `{op}_test.csv` | `tests/apitest/opstest/csv/{op}_test.csv` |
| 2 | `{op}.csv` | `tests/apitest/opstest/csv/{op}.csv` |

### 执行命令：
```bash
find <ATB_REPO_PATH>/tests/apitest/opstest/csv/ -name '{op}*.csv' -type f | while read f; do
    echo "=== Running: $f ==="
    python3 atb_csv_ops_test.py -i "$f"
done

Before full regression passes entirely (Gate 5b passes), entering Phase 6 delivery is prohibited.


### Phase 5a/5b 测试策略对照

| 阶段 | 测试范围 | CSV | 执行方式 | 迭代数 |
|------|----------|-----|---------|--------|
| Phase 5a Step 1 | 设计用例（正例+反例） | `_func.csv` | **前台**，等待完成 | 默认 1 |
| Phase 5a Step 2 | 设计用例（性能） | `_perf.csv` | **后台 subagent**，逐个 `-n N:N` | 默认 400 |
| Phase 5b 全量回归 | 所有相关 CSV | `_func.csv` + `_perf.csv`（或 `_test.csv`） | 前台逐个 | 默认 1 / 400 |

---

## 详细内容

<details>
<summary>🔍 查看详细内容</summary>

### 前置条件

- CANN 环境已安装并 source
- ATB 测试框架已编译（`libatb_test_framework.so` 存在）
- ATB 源码仓库已克隆

### 执行步骤

#### 第1步：进入 Docker 容器环境

```bash
docker exec -it <DOCKER_NAME> bash

Step 2: Source CANN environment

source <CANN_PATH>/set_env.sh

Step 3: Source ATB output environment

source <ATB_REPO_PATH>/output/atb/set_env.sh

Step 4: Verify the test framework

ls -lh $ATB_HOME_PATH/lib/libatb_test_framework.so

If the file does not exist, first run the compilation:

cd <ATB_REPO_PATH>
bash scripts/build.sh testframework
source output/atb/set_env.sh

Step 5: Run the CSV test

Method 1: Use the skill script

bash <SKILLS_DIR>/atb-csv-tester/scripts/run-csv-tests.sh \
    --atb-repo-path <ATB_REPO_PATH> \
    --csv-file <CSV_FILE_PATH> \
    --case-range <CASE_RANGE>

Method 2: Run manually

cd <ATB_REPO_PATH>/tests/framework/python/CsvOpsTestTool
python3 atb_csv_ops_test.py -i <CSV_FILE_PATH>

Test run strategy

⚠️ Positive/negative cases and performance tests need to be treated differently,-t parameter behavior differs.

`-t` parameter actual behavior

atb_csv_ops_test.py Based on the TestType column, automatically determine the number of runs:

if TestType == "Performance":
    实际次数 = 400 if args.times < 400 else args.times + 1
else:
    实际次数 = args.times  # 默认 1

The first 200 iterations of performance testing skip accuracy statistics (SKIP_PERFORMANCE_CASE_NUM = 200), accumulating only from the 201st iteration onward

Recommended strategy by case type

Case type	Recommended `-t`	Actual run count	Purpose
Positive case (ExpectedError=NO_ERROR, TestType≠Performance)	Not specified (default `-t 1`）	1	Functional correctness verification
Negative case (ExpectedError≠NO_ERROR)	Not specified (default `-t 1`）	1	Error branch coverage verification
Performance development verification (TestType=Performance)	Not specified	400 (200 warm-up + 200 statistics)	Quickly verify performance and accuracy during development
Performance delivery collection (TestType=Performance)	Not specified	400	Complete collection before CI/delivery

Note：-t 50 For Performance casesdoes not take effect(50 < 400 → falls back to 400). If the number of performance iterations really needs to be reduced, you must change the TestType of the corresponding case in the CSV from Performance to empty or Function, but this will skip the performance threshold check.

Phase 5a test strategy (dual files + two-step process)

⚠️ Phase 5a must be executed in two steps: foreground functional verification → background performance testing.

Step 1: Run the functional CSV in the foreground (positive + negative cases)

cd <ATB_REPO_PATH>/tests/framework/python/CsvOpsTestTool
python3 atb_csv_ops_test.py -i <CSV_DIR>/{op}_func.csv

Do not specify -t, default 1 run, completes in seconds
Confirm that all positive cases pass and negative cases return the expected error codes
Must wait for completion before proceeding to the next step

Step 2: Run the performance CSV in a background Subagent

Use a subagent for background execution, to avoid blocking the main process and OOM:

Agent(
    description="运行 {op} 性能测试",
    prompt="""
逐个运行 {op}_perf.csv 中的性能测试用例，使用 -n 避免 OOM：

cd {ATB_REPO_PATH}/tests/framework/python/CsvOpsTestTool
source {CANN_PATH}/set_env.sh
source {ATB_REPO_PATH}/output/atb/set_env.sh

# 获取性能用例总数
lines=$(tail -n +2 {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op}_perf.csv | wc -l)

# 逐个运行每个性能 case
for i in $(seq 1 $lines); do
    echo "=== 运行性能 case $i/$lines ==="
    python3 atb_csv_ops_test.py -i {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op}_perf.csv -n $i:$i
done

完成后汇总：通过/失败/耗时。
""",
    subagent_type="general-purpose",
    run_in_background=True
)

Process timing

Phase 5a 开始
    │
    ├─ Step 1: 前台跑 _func.csv（~30秒）
    │     └─ 全部通过 ✓
    │
    ├─ Step 2: 启动后台 subagent 跑 _perf.csv（~10分钟）
    │     └─ 逐个 -n N:N 避免 OOM
    │
    └─ 前台继续: Phase 5b 全量回归 → Phase 6 交付
          │
          └─ subagent 完成 → 自动通知 → 汇总性能结果

Detailed explanation of running performance cases one by one

Use -n N:N to specify a single case to run, avoiding OOM caused by loading multiple performance cases at the same time:

# _perf.csv 含 3 个性能 case (行号 1,2,3)
python3 atb_csv_ops_test.py -i {op}_perf.csv -n 1:1  # case 1（400次迭代）
python3 atb_csv_ops_test.py -i {op}_perf.csv -n 2:2  # case 2（400次迭代）
python3 atb_csv_ops_test.py -i {op}_perf.csv -n 3:3  # case 3（400次迭代）

Method	VRAM usage	Risk
Run the entire file at once	3 cases × 400 iterations loaded simultaneously	OOM (exit 137)
One by one `-n N:N`	Only 1 case at a time	Safe

Rule：

Phase 5a positive/negative casesDo not specify -t(default 1 run), run _func.csv, quick frontend verification
Phase 5a performance test runs _perf.csv，backend subagent one by one -n N:N, main process is not blocked
Phase 5b full regression runs _func.csv + _perf.csv(or merged _test.csv), using default iteration count
After the backend subagent completes, it automatically notifies; the Agent needs to incorporate the performance results into the delivery report

CSV file format

CSV test files use the pipe character (|) as the delimiter:

Column number	Name	Description
1	CaseNum	Test case number
3	OpName	Operator name (e.g., `LinearOperation`）
4	OpParam	Operator parameters (JSON format)
21	ExpectedError	Expected error type (e.g., `NO_ERROR`）

Common CSV test files

File name	Path	Purpose
`linear.csv`	`<ATB_REPO_PATH>/tests/apitest/opstest/csv/linear.csv`	Linear operator basic test
`linear_unals.csv`	`<ATB_REPO_PATH>/tests/apitest/opstest/csv/linear_unals.csv`	Linear Unals format test

Common issues

Execution bash scripts/build.sh testframework

Execution source output/atb/set_env.sh

Confirm whether the target SoC version is in the SocVersion column of the CSV file

About this skill

Function Overview

Path Constraints

When to Invoke

Device Support Description

Execution Rules

910B/950 Difference Case Separation Requirements

Prompt After 910B Testing Is Complete

Gate 5: Full Regression Gate

Related CSV Enumeration

Gate check process

Script for when the gate blocks

After Phase 5a passes: automatically prompt to enter full regression

Step 2: Source CANN environment

Step 3: Source ATB output environment

Step 4: Verify the test framework

Step 5: Run the CSV test

Test run strategy

`-t` parameter actual behavior

Recommended strategy by case type

Phase 5a test strategy (dual files + two-step process)

Step 1: Run the functional CSV in the foreground (positive + negative cases)

Step 2: Run the performance CSV in a background Subagent

Process timing

Detailed explanation of running performance cases one by one

CSV file format

Common CSV test files

Common issues

Related documents

external-gitcode-ascend-atb-csv-tester

About this skill

Function Overview

Path Constraints

When to Invoke

Device Support Description

Execution Rules

910B/950 Difference Case Separation Requirements

Prompt After 910B Testing Is Complete

Gate 5: Full Regression Gate

Related CSV Enumeration

Gate check process

Script for when the gate blocks

After Phase 5a passes: automatically prompt to enter full regression

Step 2: Source CANN environment

Step 3: Source ATB output environment

Step 4: Verify the test framework

Step 5: Run the CSV test

Test run strategy

-t parameter actual behavior

Recommended strategy by case type

Phase 5a test strategy (dual files + two-step process)

Step 1: Run the functional CSV in the foreground (positive + negative cases)

Step 2: Run the performance CSV in a background Subagent

Process timing

Detailed explanation of running performance cases one by one

CSV file format

Common CSV test files

Common issues

Related documents

`-t` parameter actual behavior