atb-csv-tester

ATB CSV Tester

Overview

Run ATB CSV test cases to verify operator correctness. Requires the CANN environment and a compiled ATB test framework.

Path constraints

The following paths must be obtained from the user：

<CANN_PATH>: CANN installation path (e.g. /usr/local/Ascend/ascend-toolkit/latest）
<ATB_REPO_PATH>: ATB repository path

If a path is invalid, stop immediately and report an error.

When to invoke

Running ATB CSV tests (e.g. linear.csv）
Verifying the correctness of migrated operators
Operator-level functional tests in the CANN environment

Device support

The current test environment supports only Ascend910B devices.

Execution rules

When executing testsRun only cases whose SocVersion includes Ascend910B(the test framework automatically matches the current device SocVersion)
When generating CSV test cases, device differences have already beendesigned separately(see atb-csv-testcase-generator multi-device test design principles)

910B/950 differential case separation requirements

⚠️ When 910B and 950 differ in any of the following aspects, they must be split into separate cases, each with its own SocVersion:

Difference dimension	Separation requirement
dtype support	Design positive and negative cases separately for the dtypes each supports
format support	Test each legal format separately
Parameter constraints	Verify each legal/illegal parameter value separately
Dimension limits	Design boundary values/UB upper limits separately for each
ExpectedError	The same input may trigger different error codes; design negative cases separately
Performance benchmarks	Mark performance tests with their respective SocVersion

It is strictly forbidden to merge cases with functional differences into Ascend910B;Ascend950— merging SocVersions is allowed only when the two devices are completely identical.

Prompt after 910B tests complete

After all Ascend910B tests pass, you must prompt the user:

"All 910B tests have been completed. The CSV test cases already include independent cases for Ascend950 (SocVersion does not include 910B). Please manually run the CSV tests on an Ascend950 device to verify 950 compatibility:"
cd <ATB_REPO_PATH>/tests/framework/python/CsvOpsTestTool
python3 atb_csv_ops_test.py -i <CSV_FILE_PATH>

The test command is the same for 950 devices; the user only needs to switch the test environment to an Ascend950 machine and re-run
If any 950 test fails, compare with the 910B results to analyze whether it is expected behavior caused by device differences

Gate 5: Full regression gate

Mandatory rule: Before entering Phase 6 delivery, all CSV files related to the target operator must be run and pass.

Related CSV enumeration

Using the target operator name (e.g. layer_norm) as the basis, enumerate all related CSVs:

find <ATB_REPO_PATH>/tests/apitest/opstest/csv/ -name '{op_name}*.csv' -type f | sort

Match rule	Description	Example (layer_norm)
Exact match `{op_name}.csv`	Main operator CSV	`layer_norm.csv`
Prefix match `{op_name}_*.csv`	Variant/extension CSV	`layer_norm_test.csv`

Gating Check Procedure

[Phase 5b 开始]
    │
    ▼
[步骤 1: 枚举相关 CSV] → find csv/ -name '{op}*.csv'
    │
    ▼
[步骤 2: 逐个运行] → python3 atb_csv_ops_test.py -i <每个 CSV>
    │
    ▼
[步骤 3: 汇总结果] → 记录各 CSV 通过/失败/跳过
    │
    ▼
[Gate 5 检查]
    ├─ 全部通过 → Gate 5 通过 → 可进入 Phase 6
    └─ 任一失败 → Gate 5 阻断 → 返回修复

Phrasing When a Gate Blocks

## Gate 5 阻断 — 全量回归未通过

| CSV 文件 | 总用例 | 通过 | 失败 | 状态 |
|---------|--------|------|------|------|
| {op}_test.csv | 31 | 31 | 0 | ✅ |
| {op}.csv | 90 | - | - | 未执行 |

**在 Gate 5 通过前，禁止进入 Phase 6 交付。**

下一步：运行未执行的 CSV 文件，修复失败用例，重新执行全量回归。

After Phase 5a Passes: Automatic Prompt to Enter Full Regression

When all Phase 5a design test cases pass, the Agent must automatically output the following prompt:

## Phase 5a 全部通过 — 请进入 Phase 5b 全量回归

Phase 5a 设计用例已全部通过（N/N）。

### Phase 5b 需执行的全量回归 CSV 列表：

| # | CSV 文件 | 路径 | 
|---|---------|------|
| 1 | `{op}_test.csv` | `tests/apitest/opstest/csv/{op}_test.csv` |
| 2 | `{op}.csv` | `tests/apitest/opstest/csv/{op}.csv` |

### 执行命令：
```bash
find <ATB_REPO_PATH>/tests/apitest/opstest/csv/ -name '{op}*.csv' -type f | while read f; do
    echo "=== Running: $f ==="
    python3 atb_csv_ops_test.py -i "$f"
done

Before the full regression has entirely passed (Gate 5b passed), it is forbidden to enter Phase 6 delivery.


### Phase 5a/5b 测试策略对照

| 阶段 | 测试范围 | CSV | 执行方式 | 迭代数 |
|------|----------|-----|---------|--------|
| Phase 5a Step 1 | 设计用例（正例+反例） | `_func.csv` | **前台**，等待完成 | 默认 1 |
| Phase 5a Step 2 | 设计用例（性能） | `_perf.csv` | **后台 subagent**，逐个 `-n N:N` | 默认 400 |
| Phase 5b 全量回归 | 所有相关 CSV | `_func.csv` + `_perf.csv`（或 `_test.csv`） | 前台逐个 | 默认 1 / 400 |

---

## 详细内容

<details>
<summary>🔍 查看详细内容</summary>

### 前置条件

- CANN 环境已安装并 source
- ATB 测试框架已编译（`libatb_test_framework.so` 存在）
- ATB 源码仓库已克隆

### 执行步骤

#### 第1步：进入 Docker 容器环境

```bash
docker exec -it <DOCKER_NAME> bash

Step 2: Source CANN Environment

source <CANN_PATH>/set_env.sh

Step 3: Source ATB Output Environment

source <ATB_REPO_PATH>/output/atb/set_env.sh

Step 4: Verify Test Framework

ls -lh $ATB_HOME_PATH/lib/libatb_test_framework.so

If the file does not exist, compile first:

cd <ATB_REPO_PATH>
bash scripts/build.sh testframework
source output/atb/set_env.sh

Step 5: Run CSV Tests

Option 1: Use Skill Script

bash <SKILLS_DIR>/atb-csv-tester/scripts/run-csv-tests.sh \
    --atb-repo-path <ATB_REPO_PATH> \
    --csv-file <CSV_FILE_PATH> \
    --case-range <CASE_RANGE>

Option 2: Manual Execution

cd <ATB_REPO_PATH>/tests/framework/python/CsvOpsTestTool
python3 atb_csv_ops_test.py -i <CSV_FILE_PATH>

Test Run Strategy

⚠️ Positive/negative cases and performance tests must be treated differently;-t the parameter behavior differs.

`-t` Actual Parameter Behavior

atb_csv_ops_test.py Based on the TestType column, automatically decide the number of runs:

if TestType == "Performance":
    实际次数 = 400 if args.times < 400 else args.times + 1
else:
    实际次数 = args.times  # 默认 1

Before a performance test, the first 200 iterations skip accuracy statistics (SKIP_PERFORMANCE_CASE_NUM = 200), and accumulation starts only from iteration 201

Recommended Strategy by Case Type

Case Type	Recommended `-t`	Actual Number of Runs	Purpose
Positive Case (ExpectedError=NO_ERROR, TestType≠Performance)	Not specified (default `-t 1`）	1	Functional correctness verification
Negative Case (ExpectedError≠NO_ERROR)	Not specified (default `-t 1`）	1	Error branch coverage verification
Performance Development Verification (TestType=Performance)	Not specified	400 (200 warm-up + 200 statistics)	Rapid performance and accuracy verification during development
Performance Delivery Collection (TestType=Performance)	Not specified	400	Full collection before CI/delivery

Note：-t 50 For Performance cases,does not take effect(50 < 400 → falls back to 400). If the number of performance iterations really needs to be reduced, the corresponding case's TestType in the CSV must be changed from Performance to empty or Function, but this will skip the performance threshold check.

Phase 5a Test Strategy (Dual Files + Two-Step Procedure)

⚠️ Phase 5a must be executed in two steps: foreground functional verification → background performance testing.

Step 1: Run the Functional CSV in the Foreground (positive + negative cases)

cd <ATB_REPO_PATH>/tests/framework/python/CsvOpsTestTool
python3 atb_csv_ops_test.py -i <CSV_DIR>/{op}_func.csv

Do not specify -t, default 1 run, completes in seconds
Confirm that all positive cases pass and negative cases return the expected error codes
Must wait for completion before proceeding to the next step

Step 2: Run the Performance CSV in the Background Subagent

Use the subagent to execute in the background, to avoid blocking the main flow and OOM:

Agent(
    description="运行 {op} 性能测试",
    prompt="""
逐个运行 {op}_perf.csv 中的性能测试用例，使用 -n 避免 OOM：

cd {ATB_REPO_PATH}/tests/framework/python/CsvOpsTestTool
source {CANN_PATH}/set_env.sh
source {ATB_REPO_PATH}/output/atb/set_env.sh

# 获取性能用例总数
lines=$(tail -n +2 {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op}_perf.csv | wc -l)

# 逐个运行每个性能 case
for i in $(seq 1 $lines); do
    echo "=== 运行性能 case $i/$lines ==="
    python3 atb_csv_ops_test.py -i {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op}_perf.csv -n $i:$i
done

完成后汇总：通过/失败/耗时。
""",
    subagent_type="general-purpose",
    run_in_background=True
)

Process Timeline

Phase 5a 开始
    │
    ├─ Step 1: 前台跑 _func.csv（~30秒）
    │     └─ 全部通过 ✓
    │
    ├─ Step 2: 启动后台 subagent 跑 _perf.csv（~10分钟）
    │     └─ 逐个 -n N:N 避免 OOM
    │
    └─ 前台继续: Phase 5b 全量回归 → Phase 6 交付
          │
          └─ subagent 完成 → 自动通知 → 汇总性能结果

Detailed Explanation of Running Performance Tests One by One

Use -n N:N to specify a single case to run, avoiding OOM caused by loading multiple performance cases simultaneously:

# _perf.csv 含 3 个性能 case (行号 1,2,3)
python3 atb_csv_ops_test.py -i {op}_perf.csv -n 1:1  # case 1（400次迭代）
python3 atb_csv_ops_test.py -i {op}_perf.csv -n 2:2  # case 2（400次迭代）
python3 atb_csv_ops_test.py -i {op}_perf.csv -n 3:3  # case 3（400次迭代）

Method	Memory Usage	Risk
Run the entire file at once	3 cases × 400 iterations loaded simultaneously	OOM (exit 137)
One by one `-n N:N`	Only 1 case at a time	Safe

Rule：

Phase 5a positive/negative casesDo not specify -t(default 1 run), run _func.csv, quick foreground validation
Phase 5a performance test runs _perf.csv，background subagent one by one -n N:N, main flow is not blocked
Phase 5b full regression runs _func.csv + _perf.csv(or merged _test.csv), using the default iteration count
The background subagent automatically notifies upon completion; the Agent must incorporate the performance results into the delivery report

CSV file format

CSV test files use the pipe character (|) as the delimiter:

Column number	Name	Description
1	CaseNum	Test case ID
3	OpName	Operator name (e.g., `LinearOperation`）
4	OpParam	Operator parameters (JSON format)
21	ExpectedError	Expected error type (e.g., `NO_ERROR`）

Common CSV test files

File name	Path	Purpose
`linear.csv`	`<ATB_REPO_PATH>/tests/apitest/opstest/csv/linear.csv`	Linear operator basic test
`linear_unals.csv`	`<ATB_REPO_PATH>/tests/apitest/opstest/csv/linear_unals.csv`	Linear Unals format test

FAQ

Execute bash scripts/build.sh testframework

Execute source output/atb/set_env.sh

Confirm whether the target SoC version is in the SocVersion column of the CSV file

About this skill

Overview

Path constraints

When to invoke

Device support

Execution rules

910B/950 differential case separation requirements

Prompt after 910B tests complete

Gate 5: Full regression gate

Related CSV enumeration

Gating Check Procedure

Phrasing When a Gate Blocks

After Phase 5a Passes: Automatic Prompt to Enter Full Regression

Step 2: Source CANN Environment

Step 3: Source ATB Output Environment

Step 4: Verify Test Framework

Step 5: Run CSV Tests

Test Run Strategy

`-t` Actual Parameter Behavior

Recommended Strategy by Case Type

Phase 5a Test Strategy (Dual Files + Two-Step Procedure)

Step 1: Run the Functional CSV in the Foreground (positive + negative cases)

Step 2: Run the Performance CSV in the Background Subagent

Process Timeline

Detailed Explanation of Running Performance Tests One by One

CSV file format

Common CSV test files

FAQ

Related documents

atb-csv-tester

About this skill

Overview

Path constraints

When to invoke

Device support

Execution rules

910B/950 differential case separation requirements

Prompt after 910B tests complete

Gate 5: Full regression gate

Related CSV enumeration

Gating Check Procedure

Phrasing When a Gate Blocks

After Phase 5a Passes: Automatic Prompt to Enter Full Regression

Step 2: Source CANN Environment

Step 3: Source ATB Output Environment

Step 4: Verify Test Framework

Step 5: Run CSV Tests

Test Run Strategy

-t Actual Parameter Behavior

Recommended Strategy by Case Type

Phase 5a Test Strategy (Dual Files + Two-Step Procedure)

Step 1: Run the Functional CSV in the Foreground (positive + negative cases)

Step 2: Run the Performance CSV in the Background Subagent

Process Timeline

Detailed Explanation of Running Performance Tests One by One

CSV file format

Common CSV test files

FAQ

Related documents

`-t` Actual Parameter Behavior