SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsatb-csv-testcase-generator

atb-csv-testcase-generator

Testing & quality

View repository

Updated 10-24 05:38

About this skill

ATB CSV Testcase Generator

Function Overview

This skill is used to generate CSV-format generalization test cases for ATB operators, supporting positive case, negative case, and performance test case design.

When to Invoke

Invoke this skill in the following situations:

  • The user needs to create CSV test cases for ATB operators
  • The user needs to generate positive, negative, or performance test cases

Task Breakdown

Type Description Suggested Quantity
Positive cases Verify normal functionality, cover typical scenarios 5-15
Negative cases Cover all parameter validation branches 10-20
Performance tests Large shape performance verification 5-10

Prerequisite Knowledge for the Task

  1. Reference existing CSV files：tests/apitest/opstest/csv/ The directory contains abundant reference cases
  2. Performance test marker：TestType Fill in the Performance column to mark it as a performance test
  3. Expected error marker：ExpectedError Fill in the error type in the NO_ERROR、I:ERROR_INVALID_TENSOR_DIM column, such as

Data Type Coverage

  • float16 (most commonly used)
  • bf16 (high-precision scenarios)

Shape Typicality

Include common shapes from mainstream models:

  • Llama: 512, 7168 (hidden_size=3584)
  • GPT: 256, 12288 (hidden_size=6144)
  • Mini batch: 1-16 tokens
  • Large batch: 1024-2048 tokens

Boundary Conditions

  • UB upper limit boundary (e.g., hidden_size=31424)
  • Non-aligned hidden_size (e.g., 130, 65)
  • Minimum shape (e.g., 1, 64)

Positive Case Template

CaseNum|CaseName|OpName|OpParam|InNum|InDType|InFormat|InShape|OutNum|OutDType|OutFormat|OutShape|DataGenType|DataGenRange|InTensorFile|OutTensorFile|TestType|TestLevel|FromModel|SocVersion|ExpectedError
1      |XxxBase |XxxOperation|{"param":0}|1|float16|nd|128,4096|2|int8;float|nd;nd|128,2048;128|customize|-100,100;|||||||Ascend910B;Ascend950|NO_ERROR

Data type error

# 只支持 float16/bf16，不支持 int64
SwigluQuantErrorDtypeInt64|SwigluQuantOperation|{"quantType":0}|1|int64|nd|128,4096|...|I:ERROR_INVALID_TENSOR_INI_MATCH

Dimension error

# dimNum 必须为 2
SwigluQuantErrorDimNum|SwigluQuantOperation|{"quantType":0}|1|float16|nd|128,4096,1|...|I:ERROR_INVALID_TENSOR_DIM_NUM

Dimension value error

# hidden_size 必须能被 2 整除
SwigluQuantErrorHiddenSizeOdd|SwigluQuantOperation|{"quantType":0}|1|float16|nd|128,4097|...|I:ERROR_INVALID_TENSOR_DIM

Parameter error

# quantType 只能为 0
SwigluQuantErrorParam|SwigluQuantOperation|{"quantType":1}|1|float16|nd|128,4096|...|C:ERROR_INVALID_PARAM

Shape boundary exceeded

# UB 上限检查
SwigluQuantErrorHiddenSizeTooLarge|SwigluQuantOperation|{"quantType":0}|1|float16|nd|5,15714|...|I:ERROR_INVALID_TENSOR_DIM

Negative Case Coverage Checklist

Validation Type ExpectedError Coverage Scenario
Data type I:ERROR_INVALID_TENSOR_INI_MATCH int64, int32, float32, bool
Data format I:ERROR_INVALID_TENSOR_INI_MATCH fractal_z, ncdhw, etc.
Number of dimensions I:ERROR_INVALID_TENSOR_DIM_NUM 1D, 3D, 4D
Dimension value I:ERROR_INVALID_TENSOR_DIM 0, odd numbers, out of bounds
Parameter validation C:ERROR_INVALID_PARAM Invalid parameter values
Output shape S:ERROR_INVALID_TENSOR_DIM shape mismatch
UB boundary I:ERROR_INVALID_TENSOR_DIM hidden_size too large

Large Shape Coverage

# 大 shape
SwigluQuantPerformanceLarge|SwigluQuantOperation|{"quantType":0}|1|float16|nd|2048,16384|...|Performance|Ascend910B;Ascend950|NO_ERROR

# 中 shape
SwigluQuantPerformanceMedium|SwigluQuantOperation|{"quantType":0}|1|float16|nd|512,8192|...|Performance|Ascend910B;Ascend950|NO_ERROR

# 小 shape
SwigluQuantPerformanceSmall|SwigluQuantOperation|{"quantType":0}|1|float16|nd|64,256|...|Performance|Ascend910B;Ascend950|NO_ERROR

Different data types

# bf16 性能
SwigluQuantPerformanceBf16|SwigluQuantOperation|{"quantType":0}|1|bf16|nd|1024,8192|...|Performance|Ascend910B;Ascend950|NO_ERROR

Boundary value tests

# hidden_size 边界
SwigluQuantPerformanceHidden4096|SwigluQuantOperation|{"quantType":0}|1|float16|nd|256,8192|...|Performance|Ascend910B;Ascend950|NO_ERROR

Column Description

Column Number Name Description
1 CaseNum Case number
2 CaseName Case name
3 OpName Operator name (e.g., SwigluQuantOperation）
4 OpParam JSON-format parameter string
5 InNum Number of input tensors
6 InDType Input data types (semicolon-separated)
7 InFormat Input data formats (semicolon-separated)
8 InShape Input shapes (semicolon-separated)
9 OutNum Number of output tensors
10 OutDType Output data types
11 OutFormat Output data formats
12 OutShape Output shapes
13 DataGenType Data generation type (usually customize）
14 DataGenRange Data generation range
15 InTensorFile Input tensor file (empty)
16 OutTensorFile Output tensor file (empty)
17 TestType Test type (empty or Performance）
18 TestLevel Test level (empty)
19 FromModel Source model (empty)
20 SocVersion Target SoC (e.g. Ascend910B、Ascend950. Separate multiple devices with ; :Ascend910B;Ascend950）
21 ExpectedError Expected error (NO_ERROR or error type)

Data type mapping

Use data type names supported by data_generation.py in the CSV:

Actual type Used in CSV
float32 float
float16 float16 or half
bfloat16 bf16 or bfloat16
int8 int8 or char
int32 int32 or int
int64 int64 or long

Common ExpectedError types

Error type Description Trigger scenario
NO_ERROR No error, positive case expectation Normal execution
I:ERROR_INVALID_TENSOR_INI_MATCH Tensor initialization mismatch dtype/format error
I:ERROR_INVALID_TENSOR_DIM_NUM Incorrect number of dimensions dimNum does not meet expectations
I:ERROR_INVALID_TENSOR_DIM Incorrect dimension value Dimension value out of range/invalid
C:ERROR_INVALID_PARAM Parameter error Invalid parameter value
S:ERROR_INVALID_TENSOR_DIM Incorrect output shape Output tensor shape mismatch

Multi-device test design principles

When designing CSV test cases, both Ascend910B and Ascend950 devices must be covered.

Merge vs. separate rules

Case SocVersion Description
910B and 950 fully identical(scenario/constraints/dtype/format are all the same) Ascend910B;Ascend950 Merge into one test case
910B and 950 have any difference Split into two, each with its own SocVersion Must be separated

Scenarios that must be separated (not limited to the following)

Difference type Example Handling
Different dtype support 910B supports bf16, 950 does not Split into two: 910B bf16 / 950 uses another dtype
Different format support 910B supports ND, 950 supports only ND_TRANSPOSE Split into two, each specifying a valid format
Different parameter constraints 910B quantType=0,1；950 quantType=0 Split into two, each with its own valid/invalid parameters
Different dimension limits 910B hidden_size≤31424；950 hidden_size≤32768 Design positive/boundary cases separately
Different ExpectedError For the same invalid input, 910B reports ERROR_A, 950 reports ERROR_B Split into two, each with its own ExpectedError
Different performance baselines 910B and 950 have different performance expectations Split into two, each with its own Performance

Design process

  1. First analyze the interface documentation: compare the ACLNN interface specifications of 910B and 950, and mark all differences
  2. First write common test cases: for scenarios with no differences, use Ascend910B;Ascend950
  3. Then write difference test cases: for each difference, write a separate test case for 910B and 950
  4. Difference test case naming：{OpName}{场景}910B / {OpName}{场景}950 For easy distinction

Note: When actually running tests, only Ascend910B test cases are executed (due to current test environment limitations). Ascend950 test cases are manually verified by the user on 950 devices.

⚠️ Path constraint (must be enforced)

Before executing this skill,the following paths must be obtained from the user：

  • <ATB_REPO_PATH>: ATB repository path (e.g., {your working path}ascend-transformer-boost）

Writing steps

Step 1: Obtain interface specifications

Obtain the operator's following items from the user or official documentation:

  • Input/output definitions
  • Parameter constraints
  • Data type support

Step 2: Read the source code to obtain validation logic

# 查看参数校验
cat <ATB_REPO_PATH>/src/ops/ops_infer/<op_name>/<op_name>_operation.cpp

# 查找 ERROR_INVALID_* 分支
grep -n "ERROR_INVALID" <ATB_REPO_PATH>/src/ops/ops_infer/<op_name>/<op_name>_operation.cpp

Step 3: Design positive cases

Coverage:

  • Base cases for typical data types
  • Common shapes of mainstream models
  • Boundary conditions

Step 4: Design negative cases

Cover all validation branches:

  • Data type validation
  • Dimension count validation
  • Dimension value validation
  • Parameter validation
  • Output tensor validation

Step 5: Design performance tests

Coverage:

  • Large/medium/small shapes
  • Different data types
  • Boundary values

Step 6: Create CSV files (dual-file output)

⚠️ Two independent CSV files must be produced and cannot be merged into one：

# 创建功能用 CSV（正例 + 反例）
touch <ATB_REPO_PATH>/tests/apitest/opstest/csv/<op_name>_func.csv

# 创建性能用 CSV（性能用例）
touch <ATB_REPO_PATH>/tests/apitest/opstest/csv/<op_name>_perf.csv

# 可选：创建合并全量文件
touch <ATB_REPO_PATH>/tests/apitest/opstest/csv/<op_name>_test.csv

Splitting rules

File Content TestType column Purpose
{op}_func.csv All test cases with TestType other than Performance (positive + negative cases) Empty or Function Phase 5a foreground quick validation
{op}_perf.csv All test cases with TestType=Performance Performance Phase 5a background subagent run
{op}_test.csv All test cases (func + perf merged) Mixed Full reference / Phase 5b full regression

Rationale for splitting

  • Positive/negative cases use -t 1(default) and complete in seconds
  • Performance cases default to 400 iterations, are time-consuming, and may cause OOM
  • Separate files allow Phase 5a to quickly validate functionality first, then run performance in the background
  • _func.csv and _perf.csv each have an independent complete header (21 columns)

Step 7: Automatically invoke the Reviewer Agent for review ⭐

⚠️ Mandatory step — must be executed before Gate 2

After the CSV files are created,you mustinvoke the superpowers:requesting-code-review agent for review; this cannot be skipped.

Review content:

Review dimension Check content
Completeness Whether positive/negative/performance cases cover all necessary scenarios
Correctness Whether the OpParam JSON is consistent with the parameter mapping in the design document
Shape consistency Whether InShape/OutShape is consistent with the ACLNN interface constraints
Device separation Whether the SocVersion is correctly separated for 910B/950 difference scenarios
Format specification 21 columns, `

Review input:

  • CSV file path
  • Design document path (from Phase 1's *_replacement_design.md）
  • ACLNN interface parameter constraints (from detailed design Chapter 2)

Review pass criteria:

  • No missing positive-case scenarios (dtype/shape/boundary)
  • Negative cases cover all ERROR_INVALID_* branches
  • All shapes pass the ACLNN parameter constraint pre-check
  • No format errors

After review issues are fixed, only then can you proceed to Gate 2 user confirmation.

Step 8: Run tests for verification

source <CANN_PATH>/set_env.sh
source <ATB_REPO_PATH>/output/atb/set_env.sh --cxx_abi=1
cd <ATB_REPO_PATH>/tests/framework/python/CsvOpsTestTool
python3 atb_csv_ops_test.py -i <CSV_FILE_PATH>

Before starting to write, confirm the following questions:

  1. Operator interface：

    • Number and shapes of input/output tensors?
    • Supported data types?
    • Legal value ranges of parameters?

  2. Existing references：

    • Is there a CSV of the same type of operator available for reference?
    • Are there already test cases for this operator?

  3. Special constraints：

    • Are there UB limits?
    • Are there memory limits?
    • Are there specific shape requirements?

Output template

## CSV 测试用例设计

### 算子信息
- **算子名称**: XxxOperation
- **接口规格**: [简述]

### 正例列表 (N 个)
| CaseName | InShape | OutShape | 说明 |
|----------|---------|----------|------|
| XxxBase | 128,4096 | 128,2048 | 基例 |
| ... | ... | ... | ... |

### 反例列表 (M 个)
| CaseName | ErrorType | 覆盖分支 |
|----------|-----------|----------|
| XxxErrorDtype | ERROR_INVALID_TENSOR_INI_MATCH | dtype 校验 |
| ... | ... | ... |

### 性能测试列表 (P 个)
| CaseName | InShape | 说明 |
|----------|---------|------|
| XxxPerfLarge | 2048,16384 | 大 shape |
| ... | ... | ... |

### 统计
- 正例: N 个
- 反例: M 个
- 性能测试: P 个
- 总计: N+M+P 个

### 保存路径
- **功能 CSV**: `{ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_func.csv`（正例+反例）
- **性能 CSV**: `{ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_perf.csv`（性能用例）
- **合并 CSV**（可选）: `{ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_test.csv`
- **备份/预览**: `{WORKING_DIR}/`

Human in the Loop: User confirmation checkpoint ⭐

⚠️ Mandatory checkpoint - cannot be skipped

After the CSV test case design is complete, you must wait for user confirmation before entering the actual migration phase.

Preparation before confirmation

  1. Save CSV: Save the generated CSV test cases to the ATB repository directory

    • Functional CSV: {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_func.csv(positive + negative cases)
    • Performance CSV: {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_perf.csv(performance test cases)
    • Merged CSV(optional): {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_test.csv

  2. Prepare confirmation list: for the user to confirm item by item

User confirmation checklist

## CSV 用例确认 - {OpName}

### 待确认文件
- **功能 CSV**: `{ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_func.csv`（正例 X 个 + 反例 Y 个）
- **性能 CSV**: `{ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_perf.csv`（性能 Z 个）

### 确认检查表
| 检查项 | 说明 | 用户确认 |
|--------|------|----------|
| 正例覆盖典型场景 | 包含 float16/bf16 基础用例，主流模型常见 shape | [ ] |
| 反例覆盖所有校验分支 | 覆盖所有 ERROR_INVALID_* 类型 | [ ] |
| 性能测试覆盖大小 shape | 包含大/中/小 shape 测试 | [ ] |
| CSV 已拆分为双文件 | `_func.csv` 和 `_perf.csv` 各自独立 | [ ] |
| CSV 格式符合规范 | 列数=21，分隔符=|，无多余空格 | [ ] |
| 数据类型名称正确 | 使用 data_generation.py 支持的名称 | [ ] |
| ExpectedError 类型正确 | 与参数校验逻辑匹配 | [ ] |
| 可进入实际迁移阶段 | CSV 用例已通过审查 | [ ] |

### 确认方式
- **通过**: 回复 "确认通过" 或逐项打勾确认
- **修改**: 指出具体问题，返回修改后重新确认

### 后续步骤
确认通过后，将进入 Phase 3: 实际迁移阶段
- 下一步调用: `atb-aclnn-operator-migration`

**⚠️ 未收到用户确认前，禁止进入实际迁移阶段**

How to present to the user

我已生成 {OpName} 算子的 CSV 测试用例，已拆分为两个独立文件，请确认以下内容：

📄 **功能 CSV**: `{ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_func.csv`
   - 正例: N 个
   - 反例: M 个
📄 **性能 CSV**: `{ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_perf.csv`
   - 性能测试: P 个

Phase 5a 测试时将分两步：
1. 前台跑 _func.csv（秒级完成）
2. 后台 subagent 逐个跑 _perf.csv（避免 OOM）

请检查：
1. 正例是否覆盖了所有典型场景和数据类型？
2. 反例是否覆盖了所有参数校验分支？
3. 性能测试的 shape 选择是否合理？
4. CSV 拆分是否正确（正例/反例在 _func，性能在 _perf）？

回复 "确认通过" 进入实际迁移阶段，或指出需要修改的地方。
File type Path Description
Functional CSV {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_func.csv Positive + negative cases, run in the foreground in Phase 5a
Performance CSV {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_perf.csv Performance test cases, run in the background subagent in Phase 5a
Merged CSV (optional) {ATB_REPO_PATH}/tests/apitest/opstest/csv/{op_name}_test.csv Full reference

Variable description:

  • ATB_REPO_PATH: ATB repository path
  • WORKING_DIR: working_files directory
  • op_name: operator name in lowercase (e.g. {xxx}quant）

Execution result

⚠️ Fill in after execution: After the skill execution is complete, fill in the actual execution result according to the format below.

Checkpoint checklist

Step Checkpoint description Status
- Positive case design complete ✅ PASS / ❌ FAIL / ⏭️ SKIP
- Negative case design complete ✅ PASS / ❌ FAIL / ⏭️ SKIP
- Performance test design complete ✅ PASS / ❌ FAIL / ⏭️ SKIP
- CSV files created (dual files) ✅ PASS / ❌ FAIL / ⏭️ SKIP
- CSV split into _func.csv and _perf.csv ✅ PASS / ❌ FAIL / ⏭️ SKIP
- User has confirmed the CSV test cases ⭐ ⬜ Pending confirmation

VERDICT: ✅ SUCCESS / ⚠️ PARTIAL / ❌ FAILED / ⏭️ SKIPPED

Issue list (if any)

Level Checkpoint Issue description Suggestion
🔴 CRITICAL - - -
🟡 WARNING - - -

Execution summary

  • Execution time：
  • Operator name：
  • Number of positive cases：
  • Number of counterexamples：
  • Number of performance tests：
  • Pass rate：