external-cannbot-ops-pypto-precision-compare

PyPTO Operator Precision Issue Debugging Skill

Provides two precision comparison methods for quickly locating the specific op in a PyPTO operator that causes precision issues.

Method Selection

Method Comparison

Feature	File Save Method	Binary Comparison Method
Implementation	Use pypto.pass_verify_save() to save to file, use torch.save() to save golden	Use checkpoint tensors as input parameters, compare directly in memory
Applicable Scenarios	First-time localization of precision issues, need to quickly find the problem scope	Need to compare data across multiple loop iterations, reduce file IO overhead
Loop Support	Only save idx=0 data	Supports saving data for all loop iterations
Code Modification	No need to modify the kernel function signature	Need to modify the kernel function signature, add checkpoint parameters
Data Type	Save original types directly, unify conversion during comparison	Compare directly in memory, types must be consistent
Usage Difficulty	Simple, only need to add checkpoint calls	More complex, need to manage checkpoint tensors

Selection Guide

Prefer the file save method(refer to verify.md）：

• Need to quickly find the problem scope
• Only need to compare data from a single loop, no need to compare across multiple loop iterations
• Do not want to modify the kernel function signature, keep the code concise
• Complex operators with many loops, the binary comparison method makes it difficult to move data outside the loop

Switch to the binary comparison method(refer to binary-search.md）：

• Need to compare data across multiple loop iterations (such as idx=0,1,2,...)
• When the saved data files are too large, use the binary method to compare directly in memory and reduce file IO overhead

Quick Start

Using the File Save Method

1. Read verify.md to learn the detailed steps
2. Add pypto.pass_verify_save() call in the kernel
3. Add torch.save() call in the golden
4. Run the test to generate data
5. Use the comparison tool to analyze the results

Using the Binary Comparison Method

1. Read binary-search.md to learn the detailed steps
2. Modify the kernel function signature, add checkpoint tensor parameters
3. Modify the golden function to return checkpoint data
4. Modify the test function, create checkpoint tensors and compare
5. Run the test and analyze the results

Core Principles

Data Alignment Principle

No matter which method is used, you must ensure:

• The computation logic, tiling method, and data dimensions of the golden and kernel are completely consistent
• If the implementations are inconsistent, rewrite the golden function to make it consistent with the kernel
• The positions and order of checkpoints must correspond one-to-one

Checkpoint naming principles

• Use meaningful names that reflect the computation steps
• Add numeric prefixes in computation order (e.g. 1_after_matmul, 2_after_softmax）
• Ensure naming conventions are consistent so that comparison tools can match automatically

References

• PyPTO API: docs/api/
• pass_verify_save API: docs/api/others/pypto-pass_verify_save.md