profiling-analysis-computing

Profiling Computational Bottleneck Analysis Skill

Function Overview

This Skill is used to analyze computational bottleneck issues in Ascend NPU systems. It is automatically triggered when the main analysis Skill detects that the proportion of computation time exceeds 85%. It includes a complete analysis process:

Core Analysis Steps

High-Time-Consuming Operator Filtering: From op_statistic_.csv、op_summary_.csv or kernel_details.csv files, filter the Top-N high-time-consuming operators
Pivot Table Analysis: Based on the list of high-time-consuming operators, generate a performance data pivot table, analyze the proportion of each type of instruction and bottlenecks
Operator Shape Parsing: Extract the input shapes of commonly used operators such as MatMul, providing a more intuitive perspective for performance analysis

Script Composition

op_high_time_selector.py: High-time-consuming operator filtering script
op_pivot_table_analyzer.py: Pivot table analysis script
extract_op_shapes.py: Operator shape parsing script
op_perf_analysis_combine.py: Integration script, automatically executes the complete analysis process (used when called by the main skill)

Input Parameters

High-Time-Consuming Operator Filtering Script (op_high_time_selector.py)

Parameter Name	Type	Required	Description
input_path	string	Yes	Profiling file path, the root path containing the PROF_* directory, for example: ./profiling/p-perf-huawei-05_110439_20250728062428118_ascend_pt. When the skill is triggered by the main analysis Skill, prioritize analyzing the profiling file provided by the main Skill to ensure that performance analysis uses the same source data
output_path	string	No	Output result directory, used to save the generated operator list. If not specified, an output folder will be automatically created under the input path
top_n	int	No	Number of high-time-consuming operators to select, default 3

Pivot Table Analysis Script (op_pivot_table_analyzer.py)

Parameter Name	Type	Required	Description
input_path	string	Yes	Profiling file path, the root path containing the PROF_* directory. When the skill is triggered by the main analysis Skill, prioritize analyzing the profiling file provided by the main Skill to ensure that performance analysis uses the same source data
output_path	string	No	Output result directory, used to save the analysis report. If not specified, an output folder will be automatically created under the input path
top_n	int	No	Number of high-time-consuming operators to select, default 3

Operator Shape Parsing Script (extract_op_shapes.py)

Parameter Name	Type	Required	Description
--input	string	Yes	Profiling file path or directory path, supports recursive search for kernel_details or op_analysis_details files
--output	string	No	Shape parsing result output path, supports formats such as .json, .csv, .xlsx
--op	string	No	Operator type to parse, default matmul
--html-file	string	No	Optional HTML file path, used to replace the parsed shapes into the HTML report
--pattern	string	No	Optional additional file name matching patterns, can be specified repeatedly

Usage

1. Automatically called by the main analysis Skill

This Skill is usually triggered automatically by the main analysis Skill /profiling-analysis . When the main Skill detects that the computation time proportion exceeds 85%, it automatically calls this Skill's integration script op_perf_analysis_combine.pyto execute the complete analysis workflow:

High-time-consuming operator filtering: Automatically filter the Top-5 high-time-consuming operators
Pivot table analysis: Generate a performance pivot table and analyze the proportion of each instruction type
Operator shape parsing: Automatically parse shape information of operators such as MatMul

There is no need to manually execute the subsequent steps; the system will automatically complete the entire workflow from filtering to analysis.

2. Standalone use (step by step)

Step 1: Filter high-time-consuming operators

# 基本用法
python scripts/op_high_time_selector.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output"

# 指定分析前5个高耗时算子
python scripts/op_high_time_selector.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output" --top-n 5

Step 2: Pivot table analysis and bottleneck localization

# 基本用法
python scripts/op_pivot_table_analyzer.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output"

# 指定分析前5个高耗时算子
python scripts/op_pivot_table_analyzer.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output" --top-n 5

3. Standalone use (integrated version)

# 基本用法
python scripts/op_perf_analysis_combine.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output"

# 指定分析前5个高耗时算子
python scripts/op_perf_analysis_combine.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output" --top-n 5

4. Specialized operator analysis

If a specific operator is identified in the analysis results (such as op_total_duration.csv or op_analysis_details.csv) (for example, MatMul、FIA or RmsNorm), please follow the steps below for in-depth dimension extraction analysis:

Identify the operator: Confirm in the analysis report whether the target operator exists.
View the reference documentation: In the reference/ directory, find the reference documentation for the corresponding operator (for example, MatMul Shape extraction）。
Run the extraction script: According to the instructions in the reference documentation, run the extract_op_shapes.py script and specify the parameters.

Specialized analysis example

Basic usage：

python scripts/extract_op_shapes.py --input "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output "./output/matmul_shapes.csv" --op "matmul"

With HTML shape replacement functionality：

python scripts/extract_op_shapes.py --input "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output "./output/matmul_shapes.csv" --op "matmul" --html-file "./output/op_analysis_combined.html"

For details, please refer toreference documentationthe corresponding operator link in.

Analysis content

Identify high-time-consuming operators: Identify the N operators with the highest computation time (N=3 or specified by the user)
Analyze bottlenecks of high-time-consuming operators: Based on the filtered high-time-consuming operators, analyze the proportion of each instruction type and the bottlenecks
Parse common operator shapes: Extract the input shapes of common operators such as MatMul

Preparation

Data file structure

The op_statistic_.csv、op_summary_.csv and kernel_details.csv files required for analysis are located in the following directory structure:

└─*_ascend_pt
    ├─ASCEND_PROFILER_OUTPUT
    ├─FRAMEWORK
    ├─logs
    └─PROF_*
        ├─device_*
        │  └─data
        ├─host
        │  └─data
        ├─mindstudio_profiler_log
        └─mindstudio_profiler_output  # op_statistic_*.csv、op_summary_*.csv、kernel_details.csv文件位于此目录

Detailed analysis steps

1. High-time-consuming operator filtering

Depending on the type of input file, different filtering strategies are used:

1.1 Filter using op_statistic_*.csv files

File reading: Search for and read all op_statistic_*.csv files.
Data validation: Ensure the files contain the necessary "OP Type" and "Ratio(%)" columns.
Filter high-time-consuming operators: Sort the operators in descending order based on the "Ratio(%)" column, and select the top N (N=3 or specified by the user) high-time-consuming operators.
Read detailed data: Search for and read the corresponding op_summary_*.csv or kernel_details.csv files for subsequent analysis.

1.2 Filter using op_summary_*.csv or kernel_details.csv files

File reading: Search for and read all op_summary_*.csv or kernel_details.csv files.
Data validation: Ensure that the file contains the necessary columns, such as \"OP Type\", \"Task Duration(us)\", and \"Input Shapes\" in op_summary_*.csv, or \"Type\", \"Duration(us)\", and \"Input Shapes\" in kernel_details.csv.
Operator duration statistics: Classify operators based on the \"OP Type\" (op_summary_*.csv) or \"Type\" (kernel_details.csv) column, and calculate the total duration of each operator.
Filter high-duration operators: Sort operators in descending order by total operator duration, and select the top N (N=3 or user-specified) high-duration operators.

2. Pivot table analysis

Data reading: Search for and read all op_summary_*.csv or kernel_details.csv files.
Average duration statistics: For each filtered high-duration operator, calculate its average duration under each \"Input Shapes\", and sort from high to low by average duration.
Pivot analysis: Based on the sorted results, further calculate the average value of the following columns under each \"Input Shapes\": aic_mac_ratio, aic_saclar_ratio, aic_mte1_ratio, aic_mte2_ratio, aic_fixpipe_ratio, aiv_vec_ratio, aiv_saclar_ratio, aiv_mte2_ratio, aiv_mte3_ratio.
Result output: Output in table form the mean values of all specified columns, including average duration, for each combination of \"Op Types\" and \"Input Shapes\", and mark the maximum value in each row in red.
- Output files: op_total_duration.csv, op_analysis_details.csv, op_analysis_combined.html
- Output path: a user-specified path, or automatically create an output folder under the folder where the input csv file is located

3. Operator shape parsing and replacement

When the high-duration operators include operator types that support shape parsing (such as MatMul, MatMulV2, MatMulV3, etc.), automatically perform the following steps:

Detect supported operator types: Scan the list of high-duration operators and identify all operator types that support shape parsing.
Invoke the shape parsing script: Useextract_op_shapes.pyscript to extract the operator's shape dimension information from theop_analysis_details.csvorkernel_details.csvfile:
```
python scripts/extract_op_shapes.py --input <profiling数据目录> --output <输出目录>/<op_type>_shapes.csv --op <op_type>
```
Shape parsing rules: Different shape parsing rules are used for different operator types. For detailed rule descriptions, refer to the corresponding reference document (seeOperator Shape Analysis Reference Document）
Result replacement: Replace the parsed shape dimension information with the \"Input Shapes\" column of the corresponding operator in theop_analysis_combined.htmlfile, making the output result easier to read and analyze:
- Original shape:16384,1024;1024,1024
- After replacement:M=16384, K=1024, N=1024(using MatMul as an example)

Output results

Operator duration statistics table: op_total_duration.csv - list of operators sorted by duration
High-duration operator instruction ratio pivot table: op_analysis_details.csv - instruction ratios of each operator under different input shapes
Operator performance data pivot table: op_analysis_combined.html - operator performance data pivot table with visual markers, including the parsing results of operators that support shape parsing
Operator shape parsing table: <op_type>_shapes.csv - dimension parsing results of each operator that supports shape parsing

Reference document

Operator Shape Analysis Reference Documentation

Operator Type	Reference Documentation	Description
MatMul/MatMulV2/MatMulV3	MatMul Shape Extraction	M, N, K dimension parsing rules and examples for MatMul series operators

Official Documentation：