profiling-analysis-communication

Profiling Communication Bottleneck Analysis and Slow/Fast Card Detection Skill

Feature Overview

This Skill is used to analyze communication bottleneck issues in the system and detect slow/fast card phenomena. It is automatically triggered when the main analysis Skill detects that the communication time proportion exceeds 10%. It supports in-depth analysis of communication performance in cluster environments and identifies key factors affecting performance.

This Skill uses a conditional branch workflow:

1. Run the mstt tool onceObtain the slowAffectCount value of each rank
2. Perform branch judgment based on the maximum value of slowAffectCount：
   • Branch 1: maximum value of slowAffectCount > 20
     • Calculate Z-score for statistical analysis
     • If Z-score > 0.5 → determine that a slow/fast card phenomenon exists, and guide the use of the host slow/fast card issue skill
     • Otherwise → recommend checking plog logs
   • Branch 2: maximum value of slowAffectCount ≤ 20
     • Go to the communication operator anomaly analysis branch for further analysis

This Skill contains two main analysis function modules:

1. Communication Bottleneck Analysis: Extract communication operation data from the "communication" lane of profiling, and analyze the distribution and causes of communication time consumption
2. Slow/Fast Card Detection: Detect slow/fast card phenomena in cluster environments based on the mstt tool and the Z-score statistical method

Supported Data Formats

• profiling data folder

Function Module Description

1. Communication Operator Anomaly Analysis

Handle communication operator anomalies determined by the main analysis Skill as non-slow/fast card issues, and directly recommend that users check plog logs to further analyze communication issues.

Main functions:

• Receive the non-slow/fast card communication anomaly determination from the main analysis Skill
• Prompt users to check plog logs to analyze communication anomalies
• Provide plog log checking methods and anomaly handling suggestions
• Provide general communication performance optimization suggestions

Detailed documentation: Communication Operator Anomaly Analysis

2. Slow/Fast Card Detection

Analyze slow/fast card phenomena in cluster environments, identify abnormal ranks affecting performance, and provide targeted optimization suggestions.

Main functions:

• Detect slow/fast card phenomena in cluster environments
• Analyze the impact of slow/fast cards on overall performance
• Locate the specific causes leading to slow/fast cards
• Anomaly detection based on the Z-score statistical method
• Provide targeted optimization suggestions

Detailed documentation: Slow/Fast Card Detection

Usage

1. Automatically called by the main analysis Skill

This Skill is usually automatically triggered by the main analysis Skill /profiling-analysis-profiling-main . When the main Skill detects that the communication time proportion exceeds 10%, it will automatically call the corresponding sub-skill for in-depth analysis.

2. Standalone use

According to specific needs, choose to use the corresponding sub-skill:

2.1 Use Communication Bottleneck Analysis

The Communication Bottleneck Analysis function is automatically called by the main analysis Skill, and no script needs to be executed manually.

Detailed description: Please refer to Communication Bottleneck Analysis Documentation

2.2 Use Slow/Fast Card Detection

# 检测快慢卡问题
msprof-analyze cluster -d ./profiling_data -m slow_rank -o ./result

Detailed description: Please refer to Slow/Fast Card Detection Documentation

Usage Suggestions

Automatic branch process: This Skill automatically executes the following analysis process; users do not need to manually select function modules:

1. Automatic run: Automatically triggered when the main analysis Skill detects that the communication time consumption proportion exceeds 10%
2. One-time detection: Run the mstt tool once to obtain the slow and fast card counts
3. Intelligent branching: Automatically select the analysis path based on the maximum value of slowAffectCount
4. Result output: Provide targeted analysis results and recommendations

Optimization analysis path:

1. First run the communication bottleneck analysis to obtain preliminary results
2. Take the next step according to the branching recommendations in the analysis results:
   • If determined to be a slow and fast card phenomenon → use the host slow and fast card problem skill for in-depth analysis
   • Otherwise → check the plog log to confirm whether an hccl exception exists
3. Formulate a comprehensive optimization strategy based on multi-dimensional analysis results

Dependency requirements

• Slow and fast card detection dependency: msprof_analyze tool

Notes

1. Ensure that the correct version of the slow and fast card detection tool is installed
2. Provide complete profiling data to obtain accurate analysis results
3. For large-scale clusters, the analysis may take longer
4. Analyze the results comprehensively in combination with the system architecture and application scenarios
5. When the analysis results require providing plog logs, please ensure that complete log files are collected