SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsexternal-mindstudio-cluster-fast-slow-rank-detector

external-mindstudio-cluster-fast-slow-rank-detector

Data & analytics

View repository

Updated 10-15 17:19

About this skill

Cluster Fast/Slow Card Diagnosis

1. Skill Objective

In Ascend multi-card/cluster scenarios, use the Advisor tool combined with expert rules to automatically identify performance bottleneck cards (slow cards) caused by computation, communication, or Host dispatch, and drill down to locate the microscopic root causes.

2. Diagnostic Prior Knowledge Base (Expert Rules)

It is forbidden to draw conclusions based solely on the literal meaning of a single metric; the following Huawei official diagnostic logic must be strictly followed:

  • [Host Dispatch Bottleneck (Pseudo-Fast Card)]
    • Symptom: A certain card (Rank X) has an extremely long Free Time (proportion > 10% or far exceeding the average), and Compute and Communication times are abnormally short.
    • Qualitative Assessment：Rank X is by no means a fast card; rather, it is a "slow card" causing cluster blockage. Slow CPU dispatch causes its NPU to starve (producing enormous Free Time). When it finally initiates communication, the other cards have already been waiting for a long time (long Wait on other cards), so its communication completes instantly.
    • Action: Call scripts/compare_api_stats.py, focusing on observing the duration and gap differences of dispatch/synchronization APIs such as launch、aclrtSynchronizeDevice .
  • [Pure Computation Fast/Slow Card]
    • Symptom: The Free Time of each card is generally short and uniform, but the Compute Time of a certain card is significantly greater than the average.
    • Qualitative Assessment: Computation-type slow card. If the single-operator invocation count (count) differs, it is uneven load partitioning; if the count is the same but the average duration (avg_time) surges, it is operator hardware degradation or caused by dynamic Shape.
    • Action: Call scripts/compare_op_stats.py to compare operator execution differences.
  • [Communication/Slow Link Bottleneck]
    • Symptom: The communication bandwidth of each card is far below the theoretical value (e.g., SDMA < 2GB/s).
    • Qualitative Assessment: Usually small-packet communication (ZeRO3 partitioned too finely), unaligned SDMA addresses, or hardware issues.

3. Hard Constraints (MUST DO)

  1. Execute existing scripts; strictly forbidden to reinvent the wheel: The microscopic drill-down step (Step 3)must and can onlyobtain difference data by executing in the terminal the comparison script under scripts/ . It is strictly forbidden to read CSV/DB by oneself for Diff analysis before executing the script.
  2. Trace analysis prohibited: This skill process does not include Timeline-level analysis. After Step 4 outputs the report, it ends immediately; it is forbidden to proactively read trace_view.json。

4. Standard Operating Procedure (SOP)

  1. Macroscopic health check: Mandatorily call the msprof-analyze-advisor tool with the cluster path as input to obtain the overall duration range and the "slow card analysis" matrix.
  2. Bottleneck qualitative assessment: Against the [Prior Knowledge Base], determine whether the core bottleneck belongs to Host下发慢、计算慢 or 纯通信慢, and lock onto the true "slow card RankID".
  3. Micro-level drill-down: Based on the bottleneck type, directly execute the corresponding comparison script below in the terminal (use the OP script for slow computation, and the API script for slow dispatch). Only when the automatic path discovery fails and reports an error may you retry by adding the --slow-path parameter.
  4. Output report: Output the final reply according to the following structure:
    • Diagnostic conclusion: The bottleneck type and the RankID of the truly slow card.
    • Macro-level evidence: Cite the range data from the Advisor report (such as Free Time comparison).
    • Micro-level root cause: Combined with the Top difference data output by the script (time comparison of specific APIs or operators), explain the physical cause.
    • Optimization suggestions: Provide targeted suggestions (such as core binding, splitting strategy, troubleshooting operator Shape differences, etc.).

5. Script invocation manual

The comparison scripts are uniformly stored in the scripts/ folder of this skill directory, and support automatic discovery of the cluster directory or manual specification of files (CSV preferred, DB second).

Core command template:

# 【计算类瓶颈】调用算子对比脚本（将 <本技能目录> 替换为 get_skill 返回中的路径）
python <本技能目录>/scripts/compare_op_stats.py <集群数据根目录> <慢卡RankID> <快卡RankID> [--top N]

# 【下发类瓶颈】调用 API 对比脚本
python <本技能目录>/scripts/compare_api_stats.py <集群数据根目录> <慢卡RankID> <快卡RankID> [--top N]

Parameter description:

  • cluster_pah: The cluster data root directory (contains profiler_info_{rank}.json).
  • slow_rank / fast_rank: The Rank ID of the slow card and the fast card (baseline).
  • --top N: (Optional) Output the top N entries with the largest differences; default is 20.
  • --slow-path / --fast-path: (Optional) Used to manually specify the absolute paths of the slow/fast card's *.csv or *.db when the cluster automatic discovery mechanism reports an error.
  • --json: (Optional) Output in structured JSON format.