Skill Type: Evaluation type (test generation + execution + report output)
This skill performs systematic precision evaluation on compiled and installed AscendC operators. The test cases consist of two parts: "regular shape tests" and "boundary value tests". Each test case iterates over all dtypes supported by the operator, and after running, outputs a structured precision report.
Prerequisites
- The operator has been compiled and installed (the
ascend_kernelpackage can be imported or the.sofile exists) - The operator has passed basic functional tests (
tests/test_<op_name>.pyexists and passes) - The operator name, PyTorch invocation method, input domain constraints, and all supported dtypes are known
csrc/ops/<op_name>/test/<op_name>-test-cases.mdhas been generated byascendc-operator-testcase-gen(including SUPPORTED_DTYPES, TEST_SHAPES, BOUNDARY_VALUES, and operator benchmark)
Core Workflow
Phase 1: 加载用例文档 + 信息收集 → Phase 2: 用例适配 → Phase 3: 测试脚本生成 → Phase 4: 执行 → Phase 5: 报告生成
Phase 1: Load case document + information collection
Step 1.1: Load the testcase-gen case document (MANDATORY)
MUST First read csrc/ops/<op_name>/test/<op_name>-test-cases.mdand extract from it:
| Extraction Item | Location in the case document | Purpose |
|---|---|---|
| SUPPORTED_DTYPES | §Test Configuration | List of dtypes iterated over in precision testing |
| TEST_SHAPES | §Test Configuration | List of shapes for regular shape tests |
| BOUNDARY_VALUES | §Test Configuration | List of scalar values for boundary value tests |
| NPU invocation method | §Operator Benchmark | NPU_CALL expression |
| CPU reference implementation | §Operator Benchmark | CPU_REF expression |
If
<op_name>-test-cases.mddoes not exist: fall back to designing cases yourself (following the original Phase 2 workflow), but the report must note "cases are self-designed, not produced by testcase-gen".
Step 1.2: Supplement information from the code
Supplement and extract the following information from the existing code:
| Information | Source | Example |
|---|---|---|
| Operator name | User input / op_host file name | acosh |
| NPU invocation method | register.cpp in m.def(cross-validated with the case document) |
torch.ops.npu.acosh(x) |
| CPU reference implementation | PyTorch standard library (cross-validated with the case document) | torch.acosh(x.cpu().float()).to(dtype) |
| Input domain constraints | Mathematical definition / design.md | x >= 1.0 |
| All supported dtypes | op_host TORCH_CHECK in (cross-validated with the case document) |
[torch.float16, torch.float32] |
| Supported dimension/shape constraints | Logic in design.md / op_host | elementwise supports arbitrary dimensions |
| Precision threshold | Open-source precision standards for ecosystem operators (see table below) | Look up Threshold by dtype |
Precision standards (open-source precision standards for ecosystem operators)
Adopt MERE(mean relative error) and MARE(maximum relative error):
相对误差 = abs(actual - golden) / (abs(golden) + 1e-7)
MERE = mean(相对误差)
MARE = max(相对误差)
Introduce into the denominator
1e-7to avoid division by zero when golden is zero.
Acceptance criteria：MERE < Threshold and MARE < 10 × Threshold。
| dtype | Threshold | MERE upper limit | MARE upper limit (10×) |
|---|---|---|---|
| float16 | 2⁻¹⁰ ≈ 9.77e-4 | 9.77e-4 | 9.77e-3 |
| bfloat16 | 2⁻⁷ ≈ 7.81e-3 | 7.81e-3 | 7.81e-2 |
| float32 | 2⁻¹³ ≈ 1.22e-4 | 1.22e-4 | 1.22e-3 |
For the full dtype list (including HiFLOAT32, FLOAT8 E4M3/E5M2), see
references/OPS_PRECISION_STANDARDS.md。
Phase 2: Test case adaptation
Prioritize reusing testcase-gen output: if Phase 1 successfully loaded
<op_name>-test-cases.md, then TEST_SHAPES and BOUNDARY_VALUES are used directlyas defined in the test case document, with no need to redesign. Only design them yourself when the test case document does not exist.
Design principles
- Full dtype coverage: every shape / every boundary value iterates over all dtypes supported by the operator
- shape is determined by the operator: choose appropriate shapes based on the dimensions supported by the operator; do not use fixed dimensions
- shape should not be too large: keep the number of elements per test case within a reasonable range to avoid unnecessary large tensors
- Total number of test cases = (len(TEST_SHAPES) + len(BOUNDARY_VALUES)) × len(SUPPORTED_DTYPES) ≥ 30
Part A: Regular shape tests (TEST_SHAPES)
Based on the dimensions supported by the operator, select suitable shapes from the following dimension pool to form the TEST_SHAPES list.
MUST Choose according to the dimensions actually supported by the operator; do not select unsupported dimensions.
Shape selection reference pool
| Dimensions | Recommended shape | Applicable operator types |
|---|---|---|
| 1D | (128,), (1024,), (4096,), (8192,) | elementwise, reduction |
| 2D | (32, 512), (64, 768), (128, 1024) | elementwise, matmul, linear |
| 3D | (8, 16, 64), (4, 128, 256) | elementwise, attention, conv1d |
| 4D | (4, 8, 32, 16), (2, 64, 32, 32) | conv2d, elementwise |
| 5D | (2, 3, 4, 5, 6) | conv3d, elementwise |
shape should not be too large: recommended number of elements per shape ≤ 200K.
TEST_SHAPES format
TEST_SHAPES = [
("category_name", "description", (dim0, dim1, ...)),
# ...
]
Category names are user-defined; it is recommended to name them by dimension or scenario. Example:
# elementwise 算子（支持任意维度）
TEST_SHAPES = [
("1D", "128 elements", (128,)),
("1D", "1024 elements", (1024,)),
("1D", "4096 elements", (4096,)),
("1D", "8192 elements", (8192,)),
("2D", "batch*hidden 32x512", (32, 512)),
("2D", "BERT-base 64x768", (64, 768)),
("2D", "BERT-large 128x1024", (128, 1024)),
("3D", "8x16x64", (8, 16, 64)),
("3D", "4x128x256", (4, 128, 256)),
("4D", "4x8x32x16", (4, 8, 32, 16)),
("Production", "ViT 8x197x768", (8, 197, 768)),
("Production", "single sample 1x512", (1, 512)),
]
Part B: Boundary value tests (BOUNDARY_VALUES)
For the operator's input domain, select key boundary points and typical values. Use a fixed small shape (1024,) for testing.
MUST Determine boundary values based on the operator's mathematical definition; different operators vary greatly.
Boundary value design guidance
| Operator type | Recommended boundary values |
|---|---|
| acosh (x≥1) | x=1.0, x=1.001, x=10.0, x=1000.0 |
| log (x>0) | x=0.001, x=1.0, x=100.0, x=10000.0 |
| sigmoid (full domain) | x=0.0, x=-5.0, x=5.0, x=-20.0, x=20.0 |
| sqrt (x≥0) | x=0.0, x=0.001, x=1.0, x=10000.0 |
| no domain restriction | x=0.0, x=1.0, x=-1.0, x=100.0 |
BOUNDARY_VALUES format
BOUNDARY_VALUES = [
("description", scalar_value),
# ...
]
Example (acosh):
BOUNDARY_VALUES = [
("domain lower bound x=1.0", 1.0),
("near boundary x=1.001", 1.001),
("moderate value x=10.0", 10.0),
("large value x=1000.0", 1000.0),
]
Phase 3: Test script generation
MUST First read templates/ the template files in the directory, replace the placeholders, and generate them into the operator directory's test/ subdirectory.
Output directory
csrc/ops/<op_name>/test/
├── test_<op_name>_precision.py ← pytest 测试
├── run_<op_name>_precision_report.py ← 报告生成器
├── <op_name>_precision_report.json ← JSON 报告（执行后生成）
└── <op_name>_precision_report.md ← Markdown 报告（执行后生成）
Template files
| Template | Path | Generation target |
|---|---|---|
| pytest tests | templates/test_op_precision_template.py |
csrc/ops/<op_name>/test/test_<op_name>_precision.py |
| Report generator | templates/run_precision_report_template.py |
csrc/ops/<op_name>/test/run_<op_name>_precision_report.py |
Placeholder replacement table
| Placeholder | Description | Example (acosh) |
|---|---|---|
{{OP_NAME}} |
Operator name | acosh |
{{NPU_CALL}} |
NPU invocation expression (using variables x） |
torch.ops.npu.acosh(x) |
{{CPU_REF}} |
CPU reference baseline (using variables x, dtype） |
torch.acosh(x.cpu().float()).to(dtype) |
{{SUPPORTED_DTYPES}} |
List of all supported dtypes | [torch.float16, torch.float32] |
{{INPUT_LOW}} |
Domain lower bound for random input | 1.0 |
{{INPUT_HIGH}} |
Domain upper bound for random input | 11.0 |
{{TEST_SHAPES}} |
Regular shape list (output of Phase 2 Part A) | see example above |
{{BOUNDARY_VALUES}} |
Boundary value list (output of Phase 2 Part B) | See example above |
Precision metrics that must be collected
Decision metrics(used for pass/fail determination):
| Metric | Calculation method | Pass condition |
|---|---|---|
| MERE | ((npu - ref).abs() / (ref.abs() + 1e-7)).mean() |
< Threshold |
| MARE | ((npu - ref).abs() / (ref.abs() + 1e-7)).max() |
< 10 × Threshold |
Auxiliary metrics(used for analysis, not as a basis for determination):
| Metric | Calculation method | Meaning |
|---|---|---|
| MaxAbsErr | (npu - ref).abs().max() |
Maximum absolute error |
| MeanAbsErr | (npu - ref).abs().mean() |
Mean absolute error |
| CosineSim | F.cosine_similarity(npu.flatten(), ref.flatten()) |
Cosine similarity |
Phase 4: Execute tests
4.1 Environment preparation
source /usr/local/Ascend/ascend-toolkit/set_env.sh
export PATH=/root/miniconda3/envs/py310/bin:$PATH
MUST Source the environment before each shell invocation.
4.2 Execute pytest
cd <project_root>
python3 -m pytest csrc/ops/<op_name>/test/test_<op_name>_precision.py -v --tb=short
4.3 Generate report
python3 csrc/ops/<op_name>/test/run_<op_name>_precision_report.py
4.4 Failure handling
| Failure type | Troubleshooting direction |
|---|---|
| RuntimeError (NPU kernel) | Input data is outside the defined domain / NPU does not support this dtype |
| AssertionError (precision) | Check whether MERE/MARE slightly exceeds Threshold, analyze whether it is caused by boundary values |
| Individual dtype test case FAIL | Confirm whether the Threshold for this dtype matches, check whether MARE is concentrated in a few outlier points |
| A large number of FAILs | Check whether the operator Compute logic has bugs |
4.5 In-depth troubleshooting of precision issues
When a precision failure occurs (allclose does not pass, output deviation is too large, output is all zeros/NaN) and cannot be resolved by simple threshold adjustment，MUST Read and follow the ascendc-operator-precision-debug skill process to perform systematic root cause localization:
- Read
ascendc-operator-precision-debugSKILL.md - Execute according to its five-phase process: error analysis → code review → experiment isolation → instrumentation localization → fix verification
- After fixing, rerun the complete precision test of this skill and confirm that all pass
Note: only invoke when the precision issue cannot be resolved by adjusting the threshold. For cases where an individual dtype slightly exceeds the threshold due to hardware precision characteristics, prioritize resolving it by relaxing the threshold (and explaining this in the report).
Phase 5: Report generation
5.1 Markdown report
MUST Generate csrc/ops/<op_name>/test/<op_name>_precision_report.md, refer to templates/precision_report_template.md。
The report includes:
- Overview table: total cases/passed/failed/pass rate
- Precision threshold standard table
- Regular Shape test result table(grouped by category)
- Boundary value test result table
- Summary statistics by dtype
- Key findings(≥3 conclusions)
5.2 Completion prompt (file + conversation)
- File：MUST Generate
csrc/ops/<op_name>/test/<op_name>_precision_report.md(and in the same directory*_precision_report.jsonif the script outputs it), and provide the user with thefull path：
精度验证报告已生成：
csrc/ops/<op_name>/test/<op_name>_precision_report.md
csrc/ops/<op_name>/test/<op_name>_precision_report.json # 若存在
- Current conversation：MUST At the same time comply with the next section "Display results in conversation"; do not output only the path.
Display results in conversation (MANDATORY)
After pytest and the report script have finished executing and the Markdown/JSON has been generated, the assistant in thecurrent conversation's reply MUST：
- Paste readable conclusions(the user can grasp the results without opening files):
- Overview: total number of cases, number passed, number failed,pass rate(percentage).
- If there are failures: list the failed case identifiers (case name / shape / dtype / category), along with the main error metrics (such as MaxAbsErr) or the pytest summary line.
- If all pass: explicitly state "all passed" and the total number of cases.
- Key findings ≥3 items: may be consistent with the "Key Findings" in the report or distilled from the report (dtype differences, boundary value behavior, whether thresholds were tightened/relaxed, etc.).
- Optional: a summary table of pass status by dtype (excerpt; when there are many cases, only the summary rows may be listed).
- Criteria: in one or two sentences, explain the precision standard used (MERE/MARE, the open-source precision standard for ecosystem operators) and the Threshold value for each dtype.
- Path last: after displaying the above content, then attach
<op_name>_precision_report.md(and JSON) full path.
NEVER: reply only with "report generated" and the path; NEVER use "please open the Markdown yourself" as a substitute for displaying the pass rate and failure summary in the conversation.
Lessons Learned
Input Generation
- Domain: be sure to consult the mathematical definition of the operator to ensure the input is valid
- fp16 range: the fp16 maximum is about 65504; inputs must not exceed this value
- shape size: it is recommended that the number of elements per shape be ≤ 200K to avoid excessively long test times
Precision Metrics
- MERE / MARE: decision metric, denominator is
abs(golden) + 1e-7(not clamped), aligned with the open-source precision standard for ecosystem operators - MaxAbsErr / MeanAbsErr: auxiliary analysis, helps determine the magnitude of deviation
- CosineSim: when the output is all zeros it is 0 or NaN; this should be noted and explained rather than judged as a failure
Threshold Description
- Threshold source: open-source precision standard for ecosystem operators (
references/OPS_PRECISION_STANDARDS.md） - Pass condition: MERE < Threshold and MARE < 10 × Threshold
- Do not arbitrarily relax the threshold; if relaxation is truly needed,MUST explain the reason in the report
Anti-patterns (NEVER)
- NEVER generate only the report file without displaying the overview and conclusions in the conversation
- NEVER conceal the number of failed cases and report only the path
Checklist
- has read
csrc/ops/<op_name>/test/<op_name>-test-cases.md(if present) - information collection complete (operator name, invocation method, input domain, all supported dtypes, supported dimensions)
- TEST_SHAPES preferentially come from the testcase-gen case document, and shapes are not too large
- BOUNDARY_VALUES are designed according to the operator input domain
- total number of cases = (shapes + boundary) × dtypes ≥ 30
- every dtype supported by the operator has been tested
- pytest all pass
- JSON + Markdown reports have been generated
- Key findings ≥ 3 items
- the user has been prompted with the report and JSON paths (if generated)
- displayed in the current conversationoverview (pass rate), failure summary (if any), and ≥3 key findings, not just the path