Overview
This skill provides a complete workflow for PyPTO operator performance tuning, including precision verification, performance data collection, performance analysis, and iterative tuning.
Core Principles
⚠️⚠️⚠️ Very important: all tuning verification must be executed on the board; reject theoretical guessing and fabrication from thin air!!!
1. Prerequisites for performance tuning (highest priority)
⚠️ Very important: performance tuning must be built on the basis of correct precision!
⛔ Prohibited: without a record of precision verification passing, entering any tuning step is absolutely prohibited!
Core requirements：
- ✅ Precision must pass: first ensure the operator precision verification passes before performance tuning can be performed
- ✅ Verify precision every time: after each tuning modification, precision must be re-verified (do not be afraid of the trouble!)
- ❌ Do not fix precision failures：
- First-round failure: do not fix it; you may try changing cards, and if it fails multiple times, have the user confirm
- Failure caused by tuning modification: after a simple analysis, if it cannot be resolved, roll back the modification, record the failure reason, and try other optimization plans
- ⚠️ Precision issues are operator implementation issues, not something tuning can solve; you may try, but it is not mandatory to solve them
Detailed handling process: see Step 1.4 (precision verification) and Step 4.3 (iterative tuning process)
2. Iterative optimization principle
⚠️ Important: performance tuning is a cyclic iterative process!
- After modifying one change point, immediately verify precision
- After precision passes, immediately test performance
- Do nottest only after all changes are completed
- Compare performance data before and after the modification
- If there is performance regression or an abnormal situation such as execution timeout, try modifying; if it cannot be resolved, roll back the modification, record the failure, and try other plans
- Repeat the above process until the target performance is reached
⚠️ Note: when the performance target cannot be reached for a long time, or it is identified that there is no room for tuning, you may try redesigning the operator
3. Traceability principle
⚠️ Important: performance tuning is a traceable process!
- All tuning methods attempted and their performance behavior must be recorded in the result file
- Do not judge performance behavior yourself; as long as the gain is positive, keep it. For negative gains, roll back the code and keep a trace of the process
- For errors and failure scenarios encountered during tuning, please retain the detailed process and key error logs
4. Proactive learning principle
⚠️ Important: reject blind tuning; proactively query and proactively learn
- Reject blind, mindless trial-and-error tuning Do not guess blindly, do not modify blindly, and do not configure randomly based on feelings.
- Base it on documents / knowledge bases When encountering problems, consult official documents, authoritative materials, and classic cases.
- When encountering an unclear interface and being unsure how to use it, proactively consult the API interface documentation
Resource Library
- High-Performance Programming Practices -- Introduces many high-performance programming cases; you can refer to the high-performance approaches in them for optimization
- API Interface Documentation -- Introduces all interfaces and tuning parameter usage instructions of the entire pypto repository
5. Progress Visualization Principle
⚠️ Important: A todo list must be created when tuning starts, so that the user can clearly see the progress!
Creation Timing: Create immediately after accuracy verification passes
Todo Template：
## 📊 性能调优进度
### 目标
- 算子: [算子名称]
- 基准: [基准性能] us
- 目标: [目标性能] us (提升X倍)
### 进度
- ✅ 精度校验通过
- 🔄 [当前阶段]
- ⏸️ [待优化阶段]
### 性能记录
| 轮次 | 优化内容 | 执行时间 | 提升 |
|------|---------|---------|------|
| 基准 | - | XX us | - |
Update Timing：
- After each optimization (regardless of success or failure)
- When switching phases
- When performance improvement > 5%
- When there is no improvement for 3 consecutive times
Status Board(output every 5 rounds of optimization completed):
## 当前状态
- 性能: XX us (累计提升 XX%)
- 进度: ████████░░ XX%
- 成功率: X/Y 轮
Step 0: Determine the performance tuning goal
⚠️ Important: The performance goal must be clarified, otherwise it is impossible to determine when to stop tuning!
0.1 Ask the user for the performance goal
Must ask the user：
请明确性能调优目标：
- 需要提升几倍性能？（例如：提升 5 倍）
- 或者需要达到多少执行时间？（例如：≤5000 us）
If the user has not specified it, proactively ask; do not guess!
0.2 Calculate the specific target value
Calculate the specific target based on user input:
# 示例：用户要求提升 5 倍
原始执行时间 = 27469.66 us
目标执行时间 = 原始执行时间 / 5 = 5493.93 us
# 示例：用户要求执行时间≤5000 us
目标执行时间 = 5000 us
0.3 Set the tuning termination conditions
Automatic termination conditions：
- ✅ Reaching the performance goal (execution time ≤ target value)
- ✅ Core utilization > 80% and bubble rate < 10%
- ✅ Reaching the tuning time limit (12 hours by default)
Manual termination conditions：
- The user explicitly requests to stop
Step 1: Operator case execution and accuracy verification
1.1 Set environment variables
export TILE_FWK_DEVICE_ID=0 # 或其他可用 NPU 卡
export PTO_TILE_LIB_CODE_PATH=${ASCEND_HOME_PATH:-/usr/local/Ascend/cann}/aarch64-linux
⚠️ Execution timeout configuration：
- All operator execution commands must set timeout=300 seconds (5 minutes)
- When using the Bash tool to execute, add the
timeout: 300000parameter (unit: milliseconds)
Verify environment：
# 检查 NPU 设备
npu-smi info
# 检查路径存在
ls -la $PTO_TILE_LIB_CODE_PATH/include/pto/
1.2 Compilation strategy
⚠️ Important: For the first accuracy verification, compilation is required. ⚠️ Important: If only the operator test or impl code has been modified, run it directly; compilation is not required.
| Modification Type | Whether Compilation Is Required | Reason |
|---|---|---|
| First execution | ✅ Compilation required | The first execution requires updating the whl package |
| Operator test or impl code (*.py) | ❌ Not required | Python code takes effect immediately |
| framework code | ✅ Compilation required | C++ code needs to be recompiled |
| python/pypto directory | ✅ Compilation required | Core framework code |
Compilation command(execute only when needed):
# 执行编译
python3 build_ci.py -f python3 --disable_auto_execute
# 设置环境变量
export PYTHONPATH=./pypto/build_out/:$PYTHONPATH
export LD_LIBRARY_PATH=./pypto/build_out/pypto/lib/:$LD_LIBRARY_PATH
1.3 Execute the operator case
python3 custom/operator_name/operator.py --run-mode npu
1.4 Accuracy verification (⛔ mandatory checkpoint)
⛔ Prohibited: This step must be completed and passed before proceeding to Step 2!
Execute accuracy verification：
python3 custom/operator_name/operator.py --run-mode npu
⛔ Mandatory check process (must be executed after each tuning modification)：
- ✅ The test case must be run
- ✅ Must see "passed/success" or similar success output; any occurrence of "time out" is a failure
- ✅ Must record the accuracy verification result (including verification time and command)
- ❌ Prohibited: assuming accuracy passed, skipping verification, using previous verification results
⛔ Mandatory recording of verification results (must be filled in)：
### 精度验证记录
- 验证时间: YYYY-MM-DD HH:MM:SS
- 验证命令: python3 xxx.py --run-mode npu
- 验证结果: ✅ 通过 / ❌ 失败
- 关键输出: [粘贴 "test passed" 或报错信息]
✅ Passed: continue to step 2 (performance data collection)
⚡ Immediately create a tuning Todo List：
## 📊 [算子名称] 性能调优进度
### 目标
- 基准性能: XX us
- 目标性能: XX us (提升X倍)
- 当前设备: NPU 卡 X
### 进度
- ✅ 精度校验通过
- 🔄 性能数据采集
- ⏸️ 开箱性能调优
- ⏸️ 深度性能调优
- ⏸️ 核内性能调优
- ⏸️ 生成调优报告
### 性能记录
| 轮次 | 优化内容 | 执行时间(us) | 变化 | 状态 |
|------|---------|-------------|------|------|
| 基准 | - | XX | - | ✅ |
Then enable performance data collection (modify debug_options)
❌ Failed: user confirms handling
Failure handling process：
-
First failure：
- ⚠️ Do not perform repair, the user needs to confirm on their own
- You can try running on a different card (change TILE_FWK_DEVICE_ID)
- Check whether the environment configuration is correct
-
Try a different card：
# 查看可用 NPU 卡 npu-smi info # 尝试其他卡 export TILE_FWK_DEVICE_ID=1 # 或其他可用卡号 python3 custom/operator_name/operator.py --run-mode npu -
Multiple failures or timeouts：
- If multiple attempts (3 attempts recommended) still fail
- or the run times out (5 minutes recommended)
- Stop tuning, and let the user confirm whether to continue
⚠️ Important note：
- Accuracy issues are problems with the operator implementation and cannot be solved by performance tuning
- Performance tuning is based on correct accuracy
- If accuracy cannot pass, the operator implementation should be fixed first
Step 2: Performance data collection
2.1 Enable performance data collection
In the operator implementation file, modify the @pypto.frontend.jit decorator:
@pypto.frontend.jit(
debug_options={"runtime_debug_mode": 1} # 启用性能数据采集
)
def kernel_function(...):
# 算子实现
pass
⚠️ Important note: when the performance tuning task ends, restore the modified switch.
2.2 Run again (no compilation needed)
If only the operator impl code was modified, just run it directly:
python3 custom/operator_name/operator.py --run-mode npu
2.3 Performance data file location
After execution, files will be generated in the output/output_*/ directory:
merged_swimlane.json- Swimlane chart data filemachine_runtime_operator_trace.json- Performance trace filebubble_analysis.log- Bubble analysis report
Step 3: Performance data analysis
⚠️ Important: the optimization suggestions in this process are used for subsequent step-by-step performance analysis and tuning; do not start optimizing here immediately!
3.1 Analyze performance
Use the perf-analyzer sub-skill to analyze performance data and generate a performance report and optimization suggestions.
# 加载性能分析技能
Read perf-analyzer/SKILL.md
3.2 View the performance report
The performance report is saved in output/output_时间戳/performance_analysis_report.md。
3.3 Establish a performance baseline
The baseline performance must be recorded：
## 基准性能（未优化）
- 执行时间: XXX us
- 核心利用率: XX%
- 气泡率: XX%
- 负载均衡度: XX%
Step 4: Step-by-step performance analysis and tuning
⚠️ Important: you must load the sub-skills in order to obtain the detailed tuning guide!
4.0 Tuning process overview
Fixed execution order：
第1步：开箱性能调优 (10%)
├─ 加载 tune-frontend 子技能
├─ 根据性能基准优化代码写法、TileShape、BLOCK_SIZE
├─ ⚠️ 不需要查看性能报告的详细分析，只需对比性能基准
└─ 快速建立性能基准
第2步：深度性能调优 (60%)
├─ 加载 tune-swimlane 子技能
├─ 查看性能报告，分析泳道图
├─ 优化调度策略：Stitch 调优、合图调优、L1Reuse 优化
└─ 基于性能报告指导优化方向
第3步：核内性能调优 (30%)
├─ 加载 tune-incore 子技能
├─ 查看性能报告，分析核内瓶颈
├─ 指令级优化、核内流水优化
└─ 特殊 Shape 处理
⚠️ Important note：
- Out-of-the-box performance tuning: no need to view the performance report's detailed analysis, but you need to compare the baseline execution time
- Deep performance tuning: need to view the performance report, analyze the swimlane chart and performance bottlenecks
- Intra-core performance tuning: need to view the performance report, analyze intra-core instructions and the pipeline
Entry and exit conditions for each stage：
| Stage | Entry conditions | Exit conditions |
|---|---|---|
| Out-of-the-box tuning | Accuracy verification passed | Performance meets target or 5 consecutive optimizations with no improvement |
| Deep tuning | Out-of-the-box tuning cannot improve further | Performance meets target or 8 consecutive optimizations with no improvement |
| Intra-core tuning | Deep tuning cannot improve further | Performance meets target or theoretical performance limit reached |
4.1 Load sub-skills
# 第1步：加载开箱性能调优指南
Read tune-frontend/SKILL.md
# 第2步：加载深度性能调优指南
Read tune-swimlane/SKILL.md
# 第3步：加载核内性能调优指南
Read tune-incore/SKILL.md
4.2 Performance problem diagnosis
⚠️ Important: Out-of-the-box performance tuning does not require viewing performance reports!
Out-of-the-box performance tuning: Optimize directly based on the performance baseline (execution time); no need to analyze detailed performance reports
Deep/intra-core performance tuning: Based on the performance analysis report, use the decision tree to select the optimization direction:
如果核心利用率低 (<50%):
├─ 检查任务粒度 → 增大TileShape
├─ 检查调度策略 → 使用L2亲和调度
└─ 检查数据访问 → 优化内存布局
如果气泡率高 (>20%):
├─ 检查Stitch配置 → 增大stitch_function_max_num
├─ 检查任务依赖 → 使用合图优化
└─ 检查循环展开 → 使用轴切块或loop_unroll
如果负载不均衡:
├─ 检查任务分配 → 调整TileShape
└─ 检查调度策略 → 调整device_sched_mode
4.3 Iterative tuning process
┌───────────────────────────────────────┐
│ 迭代调优流程（每次循环） │
├───────────────────────────────────────┤
│ │
│ 1. 选择一个优化点 │
│ └─ 根据决策树或子技能指南 │
│ │
│ 2. 修改代码 │
│ └─ 每次只修改一个参数 │
│ │
│ 3. 验证精度 ⭐ │
│ ├─ 运行测试用例 │
│ ├─ 验证用例是否通过 │
│ └─ 失败，尝试解决，不行则回退修改 │
│ │
│ 4. 测试性能 │
│ ├─ 采集性能数据 │
│ ├─ 对比基准性能 │
│ └─ 计算提升百分比 │
│ │
│ 5. 记录结果 & 更新Todo ⭐ │
│ ├─ 性能提升：保留修改 │
│ ├─ 性能下降：回退修改 │
│ └─ 更新Todo List │
│ ├─ 成功: 标记✅并记录 │
│ └─ 失败: 标记❌并说明 │
│ │
│ 6. 检查终止条件 │
│ └─ 达到目标或无法提升则停止 │
│ │
│ 7. 状态看板（每 5 轮输出） │
│ └─ 展示当前进度和性能趋势 │
│ │
└───────────────────────────────────────┘
⚠️ Important: Verify accuracy after every modification, don't be afraid of the trouble!
⚠️ Handling of accuracy verification failure：
- Failure caused by modification: Try to resolve it; if not, roll back the modification, record the reason for failure, and try other optimization solutions
- Do not attempt to fix accuracy issues; accuracy issues are operator implementation issues and cannot be solved by tuning
Key principles：
- Modify only one optimization point at a time
- Test performance and accuracy immediately after modification
- If accuracy fails, try to resolve it; if not, roll back the modification
- If performance regresses, try other optimization points
- Each stage iterates independently, and enters the next stage only after completion
- Execute the three stages in order: out-of-the-box tuning → deep tuning → intra-core tuning
- When the tuning goal has been achieved for a long time or there is no room for tuning, you can re-examine the operator implementation and redesign and develop the operator
Todo update example：
### 性能记录（持续更新）
| 轮次 | 优化内容 | 执行时间(us) | 变化 | 状态 |
|------|---------|-------------|------|------|
| 基准 | - | 79.34 | - | ✅ |
| 1 | BLOCK_SIZE_KV 128→64 | 68.54 | -13.6% | ✅ |
| 2 | unroll_list [8,4,2,1] | 74.44 | +8.6% | ❌回退 |
| 3 | cube_nbuffer {0:4} | 66.14 | -3.5% | ✅ |
### 当前进度
- 性能: 66.14 us (累计提升 16.6%)
- 进度条: ████████░░░░░░░░ 42%
- 成功率: 2/3 轮
Status board example(output every 5 rounds):
## 📊 调优状态看板
当前阶段: 深度性能调优
性能: 66.14 us | 累计提升: 16.6% | 目标: 39.67 us
进度: ████████░░░░░░░░ 42%
### 优化统计
- 总轮次: 10 轮
- 成功: 4 轮 (40%)
- 失败: 6 轮 (60%)
- 最佳优化: 增加并行度 (+13.6%)
### 当前瓶颈
1. ⭐⭐⭐ AIC核心利用率低 (13.63%)
2. ⭐⭐⭐ 气泡率高 (59.58%)
4.4 Exception handling
3 consecutive optimizations with no improvement：
## ⚠️ 调优进展停滞
已连续 3 次优化无提升，可能原因：
1. 已达性能上限
2. 当前优化方向不佳
3. 需要算法层面改进
建议：
- 继续尝试其他优化
- 接受当前性能
- 重新设计算子
Accuracy verification failed：
## ❌ 精度验证失败
- 失败轮次: 第X轮
- 当前优化: [优化内容]
- 处理: 尝试解决，但未生效，回退修改
Todo 更新: 标记当前优化为❌
Step 5: Generate performance tuning report
5.1 Report template
⚠️ Important: After tuning ends, a tuning report must be generated!
Operation steps：
- Output the final status board：
## 📊 最终调优状态
### 目标达成
- 目标: 提升 1 倍 (XX → XX us)
- 实际: 提升XX% (XX → XX us)
- 达成率: XX%
- 状态: ✅达标 / ❌未达标
### 优化总结
- 总轮次: X轮
- 成功率: X%
- 耗时: XX分钟
- 最佳优化: [优化项] (+XX%)
### 性能趋势
基准 → 最终: XX us → XX us
-
Fill in the template content based on the actual tuning process
- Tuning overview: operator name, tuning goal, actual achievement, tuning duration, tuning rounds
- Performance comparison: original performance, final performance, improvement percentage
- Key optimizations: list all effective optimizations and their improvement ratios
- Best configuration: final optimized configuration code
- Tuning record: detailed record of each tuning round
-
Record in detail the error issues skipped during the tuning process, to facilitate subsequent resolution by developers
-
Save the tuning report
Report example：
# Flash Attention Score 性能调优报告
## 调优概述
- 算子名称: Flash Attention Score
- 调优目标: 提升 5 倍
- 实际达成: 提升 5.89 倍 (83.0%)
- 调优时长: 73 分钟
- 调优轮次: 11 轮
## 性能对比
| 指标 | 原始性能 | 最终性能 | 提升 |
|------|---------|----------|------|
| 执行时间 (us) | 27469.66 | 4665.66 | 83.0% |
| 核心利用率 | 46% | 85% | +39% |
| 气泡率 | 54% | 15% | -39% |
**性能倍数**: 5.89 倍
## 问题记录
Common errors
Error 1: Skipping sub-skills and tuning directly
Error manifestation：
- After completing step 3 (performance data analysis), directly start trying to optimize
- Did not load the corresponding sub-skill to obtain detailed guidance
- Tune by experience or guesswork
Correct approach：
- Determine the tuning direction based on the performance analysis report (refer to the decision process in step 4.2)
- Read the SKILL.md file of the corresponding sub-skill
- Perform tuning according to the detailed guidelines in the sub-skill
- Strictly follow the iterative process of "modify one place → test → verify"
Mistake 2: Modifying multiple optimization points at once
Manifestation of the mistake：
- Modify multiple configuration parameters at the same time
- Test performance only after all modifications are complete
Correct approach：
- Modify only one optimization point at a time
- Test immediately after modification
- Verify accuracy and performance
- Record the result before trying the next optimization point
Mistake 3: Not verifying accuracy
Manifestation of the mistake：
- Test performance directly after modifying the code without verifying accuracy
- Believe that small changes will not affect accuracy
Correct approach：
- Verify accuracy on every modification; don't be afraid of the trouble!
- Even for small changes, verify accuracy
- If accuracy fails, do not roll back immediately; perform a simple analysis and attempt first, and if it still cannot be resolved, then roll back
Reference materials
Sub-skills
- perf-analyzer - Performance analysis
- tune-frontend - Out-of-the-box performance tuning
- tune-swimlane - Deep performance tuning
- tune-incore - In-core performance tuning