Skill Type: Evaluation type (test generation + execution + report output)
This skill performs systematic precision evaluation on a compiled and installed AscendC operator. The test cases consist of two parts: "conventional shape tests" and "boundary value tests". Each case iterates over all dtypes supported by the operator, and a structured precision report is output after running.
Prerequisites
- The operator has been compiled and installed (the
ascend_kernelpackage can be imported or the.sofile exists) - The operator has passed basic functional tests (
tests/test_<op_name>.pyexists and passes) - The operator name, PyTorch invocation method, input domain constraints, and all supported dtypes are known
csrc/ops/<op_name>/test/<op_name>-test-cases.mdhas been generated byascendc-operator-testcase-gen(including SUPPORTED_DTYPES, TEST_SHAPES, BOUNDARY_VALUES, operator benchmark)
Core Flow
Phase 1: 加载用例文档 + 信息收集 → Phase 2: 用例适配 → Phase 3: 测试脚本生成 → Phase 4: 执行 → Phase 5: 报告生成
Phase 1: Load case document + information collection
Step 1.1: Load the testcase-gen case document (MANDATORY)
MUST First read csrc/ops/<op_name>/test/<op_name>-test-cases.md, and extract from it:
| Extraction item | Location in the case document | Purpose |
|---|---|---|
| SUPPORTED_DTYPES | §Test configuration | dtype list iterated by precision testing |
| TEST_SHAPES | §Test configuration | shape list for conventional shape tests |
| BOUNDARY_VALUES | §Test configuration | scalar value list for boundary value tests |
| NPU invocation method | §Operator benchmark | NPU_CALL expression |
| CPU reference implementation | §Operator benchmark | CPU_REF expression |
If
<op_name>-test-cases.mddoes not exist: fall back to designing cases yourself (following the original Phase 2 flow), but the report must note that "the cases are self-designed, not produced by testcase-gen".
Step 1.2: Supplement information from the code
Supplement and extract the following information from the existing code:
| Information | Source | Example |
|---|---|---|
| Operator name | User input / op_host file name | acosh |
| NPU invocation method | register.cpp in m.def(cross-validated with the case document) |
torch.ops.npu.acosh(x) |
| CPU reference implementation | PyTorch standard library (cross-validated with the case document) | torch.acosh(x.cpu().float()).to(dtype) |
| Input domain constraints | Mathematical definition / design.md | x >= 1.0 |
| All supported dtypes | op_host TORCH_CHECK in (cross-validated with the case document) |
[torch.float16, torch.float32] |
| Supported dimension/shape constraints | Logic in design.md / op_host | elementwise supports arbitrary dimensions |
| Precision threshold | Ecosystem operator open-source precision standard (see table below) | Look up Threshold by dtype |
Precision standard (ecosystem operator open-source precision standard)
Adopt MERE(mean relative error) and MARE(maximum relative error) are the two metrics for determination:
相对误差 = abs(actual - golden) / (abs(golden) + 1e-7)
MERE = mean(相对误差)
MARE = max(相对误差)
Introduce in the denominator
1e-7to avoid division by zero when golden is zero.
Pass criteria：MERE < Threshold and MARE < 10 × Threshold。
| dtype | Threshold | MERE upper limit | MARE upper limit (10×) |
|---|---|---|---|
| float16 | 2⁻¹⁰ ≈ 9.77e-4 | 9.77e-4 | 9.77e-3 |
| bfloat16 | 2⁻⁷ ≈ 7.81e-3 | 7.81e-3 | 7.81e-2 |
| float32 | 2⁻¹³ ≈ 1.22e-4 | 1.22e-4 | 1.22e-3 |
For the complete dtype list (including HiFLOAT32, FLOAT8 E4M3/E5M2), see
references/OPS_PRECISION_STANDARDS.md。
Phase 2: Test case adaptation
Prioritize reusing testcase-gen outputs: if Phase 1 successfully loaded
<op_name>-test-cases.md, then TEST_SHAPES and BOUNDARY_VALUES are used directlyfrom the definitions in the test case document, with no need to redesign. Only design them yourself when the test case document does not exist.
Design principles
- Full dtype coverage: each shape / each boundary value traverses all dtypes supported by the operator
- shapes are determined by the operator: choose appropriate shapes based on the dimensions supported by the operator; do not write fixed dimensions
- shapes should not be too large: keep the number of elements per test case within a reasonable range to avoid unnecessary large tensors
- Total number of test cases = (len(TEST_SHAPES) + len(BOUNDARY_VALUES)) × len(SUPPORTED_DTYPES) ≥ 30
Part A: Regular shape tests (TEST_SHAPES)
Based on the dimensions supported by the operator, select suitable shapes from the following dimension pool to form the TEST_SHAPES list.
MUST Select based on the dimensions actually supported by the operator; do not select unsupported dimensions.
Shape selection reference pool
| Dimension | Recommended shape | Applicable operator type |
|---|---|---|
| 1D | (128,), (1024,), (4096,), (8192,) | elementwise, reduction |
| 2D | (32, 512), (64, 768), (128, 1024) | elementwise, matmul, linear |
| 3D | (8, 16, 64), (4, 128, 256) | elementwise, attention, conv1d |
| 4D | (4, 8, 32, 16), (2, 64, 32, 32) | conv2d, elementwise |
| 5D | (2, 3, 4, 5, 6) | conv3d, elementwise |
shapes should not be too large: it is recommended that the number of elements in a single shape be ≤ 200K.
TEST_SHAPES format
TEST_SHAPES = [
("category_name", "description", (dim0, dim1, ...)),
# ...
]
Category names can be customized; it is recommended to name them by dimension or scenario. Example:
# elementwise 算子（支持任意维度）
TEST_SHAPES = [
("1D", "128 elements", (128,)),
("1D", "1024 elements", (1024,)),
("1D", "4096 elements", (4096,)),
("1D", "8192 elements", (8192,)),
("2D", "batch*hidden 32x512", (32, 512)),
("2D", "BERT-base 64x768", (64, 768)),
("2D", "BERT-large 128x1024", (128, 1024)),
("3D", "8x16x64", (8, 16, 64)),
("3D", "4x128x256", (4, 128, 256)),
("4D", "4x8x32x16", (4, 8, 32, 16)),
("Production", "ViT 8x197x768", (8, 197, 768)),
("Production", "single sample 1x512", (1, 512)),
]
Part B: Boundary value tests (BOUNDARY_VALUES)
For the operator's input domain, select key boundary points and typical values. Use the fixed small shape (1024,) for testing.
MUST Determine boundary values based on the operator's mathematical definition; different operators vary greatly.
Boundary value design guidance
| Operator type | Recommended boundary values |
|---|---|
| acosh (x≥1) | x=1.0, x=1.001, x=10.0, x=1000.0 |
| log (x>0) | x=0.001, x=1.0, x=100.0, x=10000.0 |
| sigmoid (full domain) | x=0.0, x=-5.0, x=5.0, x=-20.0, x=20.0 |
| sqrt (x≥0) | x=0.0, x=0.001, x=1.0, x=10000.0 |
| no domain restriction | x=0.0, x=1.0, x=-1.0, x=100.0 |
BOUNDARY_VALUES format
BOUNDARY_VALUES = [
("description", scalar_value),
# ...
]
Example (acosh):
BOUNDARY_VALUES = [
("domain lower bound x=1.0", 1.0),
("near boundary x=1.001", 1.001),
("moderate value x=10.0", 10.0),
("large value x=1000.0", 1000.0),
]
Phase 3: Test script generation
MUST first read the template files in the templates/ directory, replace the placeholders, and generate them into the test/ subdirectory of the operator directory.
Output directory
csrc/ops/<op_name>/test/
├── test_<op_name>_precision.py ← pytest 测试
├── run_<op_name>_precision_report.py ← 报告生成器
├── <op_name>_precision_report.json ← JSON 报告（执行后生成）
└── <op_name>_precision_report.md ← Markdown 报告（执行后生成）
Template files
| Template | Path | Generation target |
|---|---|---|
| pytest test | templates/test_op_precision_template.py |
csrc/ops/<op_name>/test/test_<op_name>_precision.py |
| Report generator | templates/run_precision_report_template.py |
csrc/ops/<op_name>/test/run_<op_name>_precision_report.py |
Placeholder replacement table
| Placeholder | Description | Example (acosh) |
|---|---|---|
{{OP_NAME}} |
Operator name | acosh |
{{NPU_CALL}} |
NPU call expression (using variable x） |
torch.ops.npu.acosh(x) |
{{CPU_REF}} |
CPU reference baseline (using variable x, dtype） |
torch.acosh(x.cpu().float()).to(dtype) |
{{SUPPORTED_DTYPES}} |
List of all supported dtypes | [torch.float16, torch.float32] |
{{INPUT_LOW}} |
Lower bound of the random input domain | 1.0 |
{{INPUT_HIGH}} |
Upper bound of the random input domain | 11.0 |
{{TEST_SHAPES}} |
Regular shape list (output of Phase 2 Part A) | see the example above |
{{BOUNDARY_VALUES}} |
Boundary value list (output of Phase 2 Part B) | See example above |
Precision metrics that must be collected
Decision metrics(used for pass/fail decision):
| Metric | Calculation method | Pass condition |
|---|---|---|
| MERE | ((npu - ref).abs() / (ref.abs() + 1e-7)).mean() |
< Threshold |
| MARE | ((npu - ref).abs() / (ref.abs() + 1e-7)).max() |
< 10 × Threshold |
Auxiliary metrics(used for analysis, not as decision basis):
| Metric | Calculation method | Meaning |
|---|---|---|
| MaxAbsErr | (npu - ref).abs().max() |
Maximum absolute error |
| MeanAbsErr | (npu - ref).abs().mean() |
Mean absolute error |
| CosineSim | F.cosine_similarity(npu.flatten(), ref.flatten()) |
Cosine similarity |
Phase 4: Execute tests
4.1 Environment preparation
source /usr/local/Ascend/ascend-toolkit/set_env.sh
export PATH=/root/miniconda3/envs/py310/bin:$PATH
MUST Source the environment before each Shell invocation.
4.2 Run pytest
cd <project_root>
python3 -m pytest csrc/ops/<op_name>/test/test_<op_name>_precision.py -v --tb=short
4.3 Generate report
python3 csrc/ops/<op_name>/test/run_<op_name>_precision_report.py
4.4 Failure handling
| Failure type | Troubleshooting direction |
|---|---|
| RuntimeError (NPU kernel) | Input data exceeds the domain / NPU does not support this dtype |
| AssertionError (precision) | Check whether MERE/MARE slightly exceeds Threshold, analyze whether it is caused by boundary values |
| Individual dtype test cases FAIL | Confirm whether the Threshold for this dtype matches, check whether MARE is concentrated at a few abnormal points |
| Many FAIL | Check whether the operator Compute logic has bugs |
4.5 In-depth troubleshooting of precision issues
When precision failures occur (allclose does not pass, output deviation is too large, output is all zeros/NaN) and cannot be resolved by simple threshold adjustments，MUST read and follow the ascendc-operator-precision-debug skill process for systematic root cause localization:
- Read
ascendc-operator-precision-debugSKILL.md - Execute according to its five-stage process: error analysis → code review → experiment isolation → instrumentation localization → fix verification
- After fixing, rerun the complete precision tests of this skill to confirm all pass
Note: only call when the precision issue cannot be resolved by adjusting thresholds. For cases where individual dtypes slightly exceed the threshold due to hardware precision characteristics, prioritize resolving by relaxing the threshold (and explain in the report).
Phase 5: Report generation
5.1 Markdown report
MUST Generate csrc/ops/<op_name>/test/<op_name>_precision_report.md, refer to templates/precision_report_template.md。
The report includes:
- Overview table: total test cases/passed/failed/pass rate
- Precision threshold standard table
- Regular Shape test results table(grouped by category)
- Boundary value test results table
- Summary statistics by dtype
- Key findings(≥3 conclusions)
5.2 Completion prompt (file + conversation)
- File：MUST Generate
csrc/ops/<op_name>/test/<op_name>_precision_report.md(and in the same directory*_precision_report.jsonif the script outputs it), and provide the user with thecomplete path：
精度验证报告已生成：
csrc/ops/<op_name>/test/<op_name>_precision_report.md
csrc/ops/<op_name>/test/<op_name>_precision_report.json # 若存在
- Current conversation：MUST also comply with the next section "Display results in conversation", and must not output only the path.
Display results in conversation (MANDATORY)
After pytest and the report script have finished executing and the Markdown/JSON have been generated, the assistant in thecurrent conversation's reply MUST：
- Paste readable conclusions(the user can grasp the results without opening files):
- Overview: total test cases, passed count, failed count,pass rate(percentage).
- If there are failures: list the failed test case identifiers (case name / shape / dtype / category), as well as the main error metrics (such as MaxAbsErr) or pytest summary lines.
- If all pass: explicitly write "all passed" and the total number of test cases.
- Key findings ≥ 3: may be consistent with the "Key Findings" in the report or distilled from the report (dtype differences, boundary value behavior, whether thresholds were tightened/relaxed, etc.).
- Optional: a summary table of pass status by dtype (excerpt; when there are many cases, only summary rows may be listed).
- Criteria: explain in one or two sentences the precision standard used (MERE/MARE, ecosystem operator open-source precision standard) and the Threshold value for each dtype.
- Path last: after presenting the above content, then append
<op_name>_precision_report.md(and the JSON) full path.
NEVER: reply only with "report generated" and the path; NEVER replace showing the pass rate and failure summary in the conversation with "please open the Markdown yourself".
Lessons Learned
Input Generation
- Domain: be sure to consult the operator's mathematical definition to ensure the input is valid
- fp16 range: fp16 maximum is about 65504, input must not exceed this value
- shape size: it is recommended that the number of elements in a single shape ≤ 200K to avoid excessively long test time
Precision Metrics
- MERE / MARE: decision metric, denominator is
abs(golden) + 1e-7(not clamp), aligned with the ecosystem operator open-source precision standard - MaxAbsErr / MeanAbsErr: auxiliary analysis, helps determine the order of magnitude of the deviation
- CosineSim: is 0 or NaN when the output is all zeros, needs to be annotated for explanation rather than judged as a failure
Threshold Explanation
- Threshold source: ecosystem operator open-source precision standard (
references/OPS_PRECISION_STANDARDS.md） - Pass condition: MERE < Threshold and MARE < 10 × Threshold
- It is not recommended to relax the threshold arbitrarily; if relaxation is truly needed,MUST explain the reason in the report
Anti-patterns (NEVER)
- NEVER only generate the report file without showing the overview and conclusions in the conversation
- NEVER conceal the number of failed test cases and only report the path
Checklist
- have read
csrc/ops/<op_name>/test/<op_name>-test-cases.md(if it exists) - information collection complete (operator name, invocation method, input domain, all supported dtypes, supported dimensions)
- TEST_SHAPES preferentially come from the testcase-gen case document, shapes are not too large
- BOUNDARY_VALUES designed according to the operator input domain
- total number of test cases = (shapes + boundary) × dtypes ≥ 30
- every dtype supported by the operator has been tested
- pytest all passed
- JSON + Markdown reports have been generated
- key findings ≥ 3
- the user has been informed of the report and JSON paths (if generated)
- shown in the current conversationoverview (pass rate), failure summary (if any), and ≥3 key findings, not just the path