profiling-analysis-profiling-computing

Profiling Compute Bottleneck Analysis Skill

Function Overview

This Skill is used to analyze compute bottleneck issues in Ascend NPU systems. It is automatically triggered when the main analysis Skill detects that the compute time proportion exceeds 85%. It contains two independent scripts:

op_high_time_selector.py: Filter high-time-consuming operators from op_statistic_.csv、op_summary_.csv or kernel_details.csv files
op_pivot_table_analyzer.py: Generate a pivot table based on the list of high-time-consuming operators

An integrated version is also providedop_perf_pivot_table.py, used to maintain backward compatibility and automatically call the above two scripts.

extract_op_shapes.py: Extract operator shape information from the op_analysis_details.csv file

Input Parameters

High-Time-Consuming Operator Filtering Script (op_high_time_selector.py)

Parameter Name	Type	Required	Description
input_path	string	Yes	Profiling file path, the root path containing the PROF_* directory, for example: ./profiling/p-perf-huawei-05_110439_20250728062428118_ascend_pt. When the skill is triggered by the main analysis Skill, prioritize analyzing the profiling file provided by the main Skill to ensure that performance analysis uses data from the same source
output_path	string	No	Output result directory, used to save the generated operator list. If not specified, an output folder will be automatically created under the input path
top_n	int	No	Number of selected high-time-consuming operators, default 3

Pivot Table Analysis Script (op_pivot_table_analyzer.py)

Parameter Name	Type	Required	Description
input_path	string	Yes	Profiling file path, the root path containing the PROF_* directory. When the skill is triggered by the main analysis Skill, prioritize analyzing the profiling file provided by the main Skill to ensure that performance analysis uses data from the same source
output_path	string	No	Output result directory, used to save the analysis report. If not specified, an output folder will be automatically created under the input path
top_n	int	No	Number of selected high-time-consuming operators, default 3

Usage

1. Automatically called by the main analysis Skill

This Skill is usually automatically triggered by the main analysis Skill /profiling-analysis-profiling-main . When the main Skill detects that the compute time proportion exceeds 85%, it will automatically call this Skill for in-depth analysis.

2. Used separately (step by step)

Step 1: Filter high-time-consuming operators

# 基本用法
python scripts/op_high_time_selector.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output"

# 指定分析前5个高耗时算子
python scripts/op_high_time_selector.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output" --top-n 5

Step 2: Pivot table analysis and bottleneck localization

# 基本用法
python scripts/op_pivot_table_analyzer.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output"

# 指定分析前5个高耗时算子
python scripts/op_pivot_table_analyzer.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output" --top-n 5

3. Used separately (integrated version)

# 基本用法
python scripts/op_perf_pivot_table.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output"

# 指定分析前5个高耗时算子
python scripts/op_perf_pivot_table.py --input-path "./p-perf-huawei-05_110439_20250728062428118_ascend_pt" --output-path "./output" --top-n 5

4. Specialized operator analysis

If a specific operator is identified in the analysis results (such as op_total_duration.csv or op_analysis_details.csv), for example MatMul、FIA or RmsNorm), please follow the steps below for in-depth dimension extraction analysis:

Identify the operator: Confirm in the analysis report whether the target operator exists.
View the reference document: In reference/ directory, find the reference document for the corresponding operator (for example MatMul Shape extraction）。
Run the extraction script: According to the instructions in the reference document, run the extract_op_shapes.py script and specify the --op parameter.

For details, please refer toReference documentthe corresponding operator link in.

Analysis content

Identify high-time-consuming operators: Identify the N operators with the highest computation time (N=3 or specified by the user)
Analyze bottlenecks of high-time-consuming operators: Based on the selected high-time-consuming operators, analyze the proportion and bottleneck of each type of instruction

Preparation

Data file structure

The op_statistic_ required for analysis.csv、op_summary_.csv and kernel_details.csv files are located in the following directory structure:

└─*_ascend_pt
    ├─ASCEND_PROFILER_OUTPUT
    ├─FRAMEWORK
    ├─logs
    └─PROF_*
        ├─device_*
        │  └─data
        ├─host
        │  └─data
        ├─mindstudio_profiler_log
        └─mindstudio_profiler_output  # op_statistic_*.csv、op_summary_*.csv、kernel_details.csv文件位于此目录

Detailed analysis steps

1. Screening of high-time-consuming operators

Different screening strategies are used depending on the input file type:

1.1 Screening using op_statistic_*.csv files

File reading: Search for and read all op_statistic_*.csv files.
Data validation: Ensure the file contains the necessary "OP Type" and "Ratio(%)" columns.
Screen high-time-consuming operators: Sort operators in descending order based on the "Ratio(%)" column, and select the top N (N=3 or specified by the user) high-time-consuming operators.
Read detailed data: Search for and read the corresponding op_summary_*.csv or kernel_details.csv files for subsequent analysis.

1.2 Screening using op_summary_*.csv files

File reading: Search for and read all op_summary_*.csv or kernel_details.csv files.
Data validation: Ensure the file contains the necessary columns, such as "OP Type", "Task Duration(us)", "Input Shapes", etc.
Operator time statistics: Classify operators based on the "Op Type" column, and calculate the total time of each operator.
Screen high-time-consuming operators: Sort operators in descending order based on total operator time, and select the top N (N=3 or specified by the user) high-time-consuming operators.

2. Pivot table analysis

Data reading: Search for and read all op_summary_*.csv or kernel_details.csv files.
Pivot analysis: For each selected high-time-consuming operator, calculate the average value of the following columns under each "Input Shapes": aic_mac_ratio, aic_saclar_ratio, aic_mte1_ratio, aic_mte2_ratio, aic_fixpipe_ratio, aiv_vec_ratio, aiv_saclar_ratio, aiv_mte2_ratio, aiv_mte3_ratio.
Result output: Output the mean values of the specified columns for each combination of "Op Types" and "Input Shapes" in table form, and mark the maximum value in each row in red.
- Output files: op_total_duration.csv, op_analysis_details.csv, op_analysis_combined.html
- Output path: the path specified by the user, or automatically create an output folder under the folder where the input csv file is located

Output results

Operator time statistics table: op_total_duration.csv - list of operators sorted by time
Pivot table of instruction proportions for high-time-consuming operators: op_analysis_details.csv - instruction proportions of each operator under different input shapes
Operator performance data pivot table: op_analysis_combined.html - pivot table of operator performance data with visual markers

Reference document

MatMul Shape extraction

Official document：