Generate verification cases and benchmarks based on the operator design document (design.md), for use by subsequent UT code generation, precision verification, and performance verification skills.
Usage scenarios
In ascendc-operator-design createddesign.mdafterwards,ascendc-operator-ut-gen/ascendc-operator-precision-eval/ascendc-operator-performance-evalcalled before.
Invocation interface
Required parameters
When invoking this skill, the following parameters must be explicitly provided:
Parameter 1: Design document
- Based on the user-provided
design.mddesign cases - Must include operator logic, inputs and outputs, constraint conditions, and supported data types
- If not specified, defaults to design.md under the project path
Design process
1. Operator design document parsing
- Read the [Operator Interface], [Computation Logic], and [Reference Implementation] sections in the design document
- Based on the operator's actual computation logic as well as existing pyTorch implementations, reference documents, etc., generate the operator benchmark and understand the mutual shape constraints among the operator's inputs and outputs.
- Based on the constraints, generate typical cases for common shapes when the operator is used in model scenarios, as well as generalized cases generated according to boundary conditions.
2. Test case generation
MANDATORY: Before generating test cases, the following reference documents must be read:
- Required reading:
templates/test-cases-template.md— Test case format referencedesign.md— Operator design document provided by the user
2.1 Design principles
- Full dtype coverage: Iterate over all dtypes supported by the operator for each shape / each boundary value,
SUPPORTED_DTYPESmust includedesign.mdall data types supported in [Parameter Description] - Shape determined by the operator: Choose appropriate shapes based on the dimensions supported by the operator; do not write fixed dimensions
- Shapes should not be too large: Keep the number of elements per case within a reasonable range to avoid unnecessarily large tensors
- Total number of cases = (len(TEST_SHAPES) + len(GENERAL_SHAPES)) × len(SUPPORTED_DTYPES) ≥ 30
Part A: Regular Shape tests (TEST_SHAPES)
Based on the dimensions supported by the operator and the operator type, select suitable shapes from the following dimension pool to form the TEST_SHAPES list.
MUST Select based on the dimensions actually supported by the operator; do not select unsupported dimensions.
Shape selection reference pool
| Dimension | Recommended shape | Applicable operator type |
|---|---|---|
| 1D | (128,), (1024,), (4096,), (8192,) | elementwise, reduction |
| 2D | (32, 512), (64, 768), (128, 1024) | elementwise, matmul, linear |
| 3D | (8, 16, 64), (4, 128, 256) | elementwise, attention, conv1d |
| 4D | (4, 8, 32, 16), (2, 64, 32, 32) | conv2d, elementwise |
| 5D | (2, 3, 4, 5, 6) | conv3d, elementwise |
Shapes should not be too large: It is recommended that the number of elements in a single shape be ≤ 200K.
TEST_SHAPES format
TEST_SHAPES = [
("category_name", "description", (dim0, dim1, ...)),
# ...
]
Category names can be customized; it is recommended to name them by dimension or scenario. Example:
# elementwise 算子（支持任意维度）
TEST_SHAPES = [
("1D", "128 elements", (128,)),
("1D", "1024 elements", (1024,)),
("1D", "4096 elements", (4096,)),
("1D", "8192 elements", (8192,)),
("2D", "32x512", (32, 512)),
("3D", "8x16x64", (8, 16, 64)),
("3D", "4x128x256", (4, 128, 256)),
("4D", "4x8x32x16", (4, 8, 32, 16)),
]
Part B: Generalized Shape tests (GENERAL_SHAPES)
Based on the dimensions supported by the operator and the operator type, select suitable shapes from the following dimension pool to form the GENERAL_SHAPES list.
MUST Select based on the dimensions actually supported by the operator; do not select unsupported dimensions.
Shape selection reference pool
Small Shape scenarios(boundary tests, extremely small values, non-aligned tests)
| Dimension | Small shape | Test purpose |
|---|---|---|
| 1D | (1,), (2,), (4,), (8,) | Minimal elements, boundary values |
| 1D | (3,), (5,), (7,) | Non-aligned lengths |
| 2D | (1, 1), (2, 2), (4, 4) | Minimal 2D tensor |
| 2D | (1, 128), (2, 256) | Single-row/single-column scenarios |
| 3D | (1, 1, 1), (2, 2, 2) | Minimal 3D tensor |
| 3D | (1, 8, 64), (2, 16, 128) | Single-batch scenario |
Large Shape scenarios(production environments, stress testing, typical model scenarios)
| Dimension | Large shape | Applicable model | Number of elements |
|---|---|---|---|
| 1D | (3072,), (4096,) | BERT FFN intermediate layer | 3K-4K |
| 1D | (5120,), (6400,) | GPT-2 FFN intermediate layer | 5K-6K |
| 2D | (512, 768) | BERT-base full sequence | 393K |
| 2D | (512, 1024) | BERT-large full sequence | 524K |
| 2D | (1024, 768) | GPT-2 sequence | 786K |
| 2D | (1024, 1024) | GPT-2 medium sequence | 1M |
| 2D | (1024, 1600) | GPT-2 XL sequence | 1.6M |
| 3D | (8, 512, 768) | BERT-base batch | 3.1M |
| 3D | (8, 197, 768) | ViT-base batch | 1.2M |
| 3D | (16, 1024, 1024) | GPT-2 large batch | 16.7M |
Notes：
- The recommended shapes for production environments and typical model scenarios should vary according to the specific application scenario of each operator.
- Small shapes are used for boundary testing to ensure the operator works correctly with extremely small inputs.
- Large shapes are used for performance testing and to verify correctness on large-scale data.
- Shapes with more than 200K elements are used only for generalization testing and are not used in regular testing.
GENERAL_SHAPES format
GENERAL_SHAPES = [
("Small", "description", (dim0, dim1, ...)),
("Large", "description", (dim0, dim1, ...)),
# ...
]
For Category, it is recommended to use "Small" and "Large" to distinguish scenarios. Example:
# elementwise 算子（支持任意维度）
GENERAL_SHAPES = [
# 小Shape场景
("Small", "single element", (1,)),
("Small", "tiny vector 2", (2,)),
("Small", "tiny vector 4", (4,)),
("Small", "unaligned length 3", (3,)),
("Small", "unaligned length 5", (5,)),
("Small", "2x2 matrix", (2, 2)),
("Small", "1x128 single row", (1, 128)),
("Small", "1x1x1 scalar", (1, 1, 1)),
("Small", "1x8x64 single batch", (1, 8, 64)),
# 大Shape场景（生产环境）
("Large", "BERT-base FFN 3072", (3072,)),
("Large", "BERT-large FFN 4096", (4096,)),
("Large", "BERT-base 512x768", (512, 768)),
("Large", "BERT-large 512x1024", (512, 1024)),
("Large", "GPT-2 1024x768", (1024, 768)),
("Large", "GPT-2 1024x1024", (1024, 1024)),
("Large", "ViT-base 8x197x768", (8, 197, 768)),
]
2.2 Design case output
Based on the collected information, read the templates/test-cases-template.md template, fill in all sections, and output to csrc/ops/[op-name]/test/[op-name]-test-cases.md。
Output location: ascend-kernel/csrc/ops/[op-name]/test/[op-name]-test-cases.md
Notes
- Strictly follow the template sections: Output only the sections defined in the template (operator benchmark, typical cases, generalization cases, usage instructions)
- Do not output irrelevant content：
- ❌ Accuracy verification standards (handled by
ascendc-operator-precision-eval) - ❌ Performance verification standards (handled by
ascendc-operator-performance-eval) - ❌ UT test code (handled by other skills)
- ❌ Any content beyond the template section definitions
- ❌ Accuracy verification standards (handled by
- Output file location：
ascend-kernel/csrc/ops/[op-name]/test/[op-name]-test-cases.md