external-cannbot-model-model-infer-parallel-analysis

Model Parallel Strategy Analysis

This skill only performs analysis and recommendations, and does not modify any code. The output is a parallel_config recommendation and solution document.

Scope of Application

Model Type：Dense Transformer、MoE Transformer
Hardware Platform: Ascend Atlas A2 / A3 series
Decision Scope：parallel_config each *_tp_size（TP）、o_proj_tp_size、cp_size（CP）、kvp_size(KVP) value, as well as the DP/EP degree derived from it (*_dp_size = world_size // *_tp_size）

Analysis Process

第一步：提取模型参数与模块链路
    ↓
第二步：定性分类 → 候选并行模式
    ↓
第三步：定量估算（先查阅 references/config-index.md 获取起点数值）
    ↓
第四步：方案审查 → 输出推荐
    ↓
（可选）第五步：Profiling 校准

Prohibited: skipping the first three steps and directly giving a configuration, or making estimates based on experience without calculation

Step 1: Extract Model Parameters and Module Links

1.1 Basic Parameters

Extract from the model config (config.json or configuration_*.py):

Parameter	Symbol	Retrieval Field	Description
Architecture Type	-	Model Documentation	Dense / MoE / MoE+MLA / MoE+DSA
Total Layers	L	`num_hidden_layers`
Number of MoE Layers	L_moe	`num_moe_layers` or inferred	0 for Dense models
Hidden size	H	`hidden_size`
FFN Intermediate Dimension	H_ffn	`intermediate_size`
Number of Attention Heads	N_h	`num_attention_heads`
Number of KV Heads	N_kv	`num_key_value_heads`	GQA: N_kv < N_h
Head dim	D_h	H / N_h or `head_dim`
Total Number of Experts	E	`num_experts`	0 for Dense models
Number of Activated Experts	E_act	`num_experts_per_tok`
Vocabulary Size	V	`vocab_size`

Additional extraction for MLA models:

Parameter	Retrieval Field	Description
KV Compression Dimension	`kv_lora_rank`	MLA's compressed KV dim
Q Compression Dimension	`q_lora_rank`	MLA's compressed Q dim
V head dim	`v_head_dim`	oproj_tp constraint:`N_h * v_head_dim % oproj_tp == 0`

1.2 Module Link Identification

Different modules have different computational characteristics in the Prefill and Decode stages, but the current analysis goal is to balance both stages undera single parallel_config . It is necessary to identify the model's key module links:

Module	Prefill Characteristics	Decode Characteristics	Parallel Dimension
Embedding	Large Vocabulary Matrix	Large Vocabulary Matrix	embed_tp
Attention QKV	Compute-intensive (large S)	Memory-access-intensive (S=1)	attn_tp / cp_size
Attention Core（FA）	Compute-intensive	Memory-access-intensive	attn_tp / cp_size / kvp_size
Attention O_proj	Compute-intensive	Memory-access-intensive	oproj_tp (set independently for MLA models)
Dense FFN	Compute-intensive	Memory-access-intensive	dense_tp
MoE Router	Lightweight	Lightweight	Follows EP
MoE Expert FFN	Compute-intensive	Distributed memory access	moe_tp (usually =1, uses EP)
LM Head	Large Vocabulary Matrix	Large Vocabulary Matrix	lmhead_tp

Key Insight: In a single-machine deployment, Prefill and Decode share one set of configurations. During analysis, determine the emphasis based on the user's target scenario: high throughput emphasizes Decode, low latency emphasizes Prefill TTFT, and balanced considers both. Do not output two separate parallel_config sets.

Note non-standard modules: The model may contain non-standard large-parameter modules (such as N-gram Embedding, multimodal Vision Encoder, etc.), whose partitioning method may differ from standard TP. These modules need to be analyzed separately for their partitioning constraints and memory usage under different parallel configurations, to avoid a plan that appears memory-feasible but actually OOMs because a certain module cannot be partitioned.

1.3 Deployment Information Confirmation

Automatically obtain：

Single-card memory M: confirm via npu-smi info or torch.npu.get_device_properties Confirm

Confirm with the user：

Parameter	Description
Number of deployment cards W	Total number of NPUs in a single node
Target scenario	High throughput / Low latency / Balanced
Maximum sequence length	Determines whether CP / KVP is needed

Current limitations: This framework only supports single-machine BF16 deployment, and does not support multi-machine distributed, PD disaggregation, or quantized inference. During analysis, TP/EP communication is all within the node, and memory is estimated as BF16.

Completion criteria

Architecture type confirmed (Dense / MoE / MoE+MLA / MoE+DSA)
Basic parameters fully extracted
Module links identified, Prefill/Decode compute characteristics annotated and incorporated into the single-configuration trade-off
Deployment information confirmed (W, M, target scenario, sequence length)
Confirmed whether there is a similar model in the repository

Step 2: Qualitative Classification

Based on the model parameters and deployment scenario, use a decision tree to determine candidate parallel modes.Prioritize aligning with best practices already verified in the repository。

The decision tree expresses the analysis approach; all judgments are baseline candidates rather than hard constraints, and must be verified with quantitative estimation. For the classification of hard constraints, strong heuristics, and implementation characteristics, see Step 4 "Constraint Check".

Unified decision tree

The decision tree has three layers: architecture backbone (determined by model structure) → scenario adjustment (determined by target scenario and batch size) → sequence length add-on (stack CP/KVP as needed). The three layers jointly produce one set of parallel_config.

Layer 1: Architecture Backbone

Determine the basic parallel method for non-Attention modules, as well as the default starting point for Attention. Layer 2 then adjusts the Attention TP degree based on the scenario.

架构类型？
├─ Dense Transformer
│  ├─ 单卡放得下？→ 基线候选：DP
│  └─ 放不下？→ 基线候选：纯 TP，节点内 TP 起步
│
└─ MoE Transformer
   │
   ├─ MoE 层
   │  ├─ EP 和 TP 均需对比（单机场景 EP 度 = 卡数）
   │  ├─ expert_ffn 较小时 TP 会导致碎矩阵，EP 计算效率更高
   │  └─ EP 需确认 experts_per_rank ≤ 24（dispatch_v2 硬件约束）
   │
   ├─ Dense FFN
   │  ├─ 有独立 Dense 层 → TP，度数控制在节点内
   │  └─ 全 MoE 无独立 Dense → dense_tp=1
   │
   ├─ O_proj（MLA 模型）→ 待第二层确定 attn_tp 后决定
   │  attn_tp>1 时 oproj_tp=1，attn_tp=1 时可独立设值
   │
   ├─ Embed/LMHead
   │  ├─ 大词表（V>100K）→ 独立设 TP
   │  └─ 小词表 → 跟随主策略
   │
   └─ 单机部署规模（卡数 W 由 1.3 确认）
      ├─ 单卡 → 无需并行
      ├─ 少卡（W < 节点满配）→ 纯 TP 为主，节点内 HCCS 带宽充足
      └─ 满卡（W = 节点满配）→ 纯 TP 或混合 EP+TP，注意 TP 碎矩阵退化

Layer 2: Scenario Adjustment

Adjust the Attention TP degree based on the target scenario and batch size.

目标场景？
├─ 高吞吐（大 batch）
│  ├─ MLA → attn_tp=1，最大化 DP，各卡独立处理不同请求
│  └─ GQA → Decode 默认 attn_tp=1 以最大化 DP；
│     Prefill 需结合 batch 和 seq_len 判断，若单卡 attention 计算成为瓶颈则保留提升 attn_tp 的候选
│
├─ 低时延（小 batch）
│  ├─ MLA → 保留小度数 attn_tp（如 2/4）作为候选，分摊 Prefill 计算降低 TTFT；
│     Decode 阶段仅在 profiling 显示 attention 计算成为明显瓶颈时才提升 attn_tp
│  └─ GQA → attn_tp 适度，分摊 Prefill 计算降低 TTFT
│
└─ 均衡
   → 在高吞吐和低时延之间折中

背景：MLA 压缩 KV 不按 head 存，TP 对 KV 侧无收益，但 Q/O 计算仍可分摊

Layer 3: Sequence Length Add-on

CP and KVP serve as add-on mechanisms, and by default are stacked without overturning the backbone of the first two layers. If their introduction changes the parallel constraints of Attention or O_proj, it is necessary to return to Layer 2 for recalibration.

最大序列长度？
├─ ≤4K → 无附加
├─ 4K-64K → 检查 Prefill 计算分摊是否充分
│  ├─ 充分 → 无附加
│  └─ 不充分
│     ├─ MLA → +CP，可能减少对大 attn_tp 的需求
│     └─ GQA → 提升 attn_tp 或 +CP
└─ 64K+ → +CP，超长 Decode 可叠加 KVP
   注意：KVP 约束 oproj_tp = kvp_size，需回到第一层校正 O_proj 配置

Switching cost between modules: Different TP degree boundaries require AllGather/ReduceScatter rearrangement, which is usually far smaller than the waste caused by a unified TP degree, but attention is needed when switching is frequent.

Quick reference table

The quick reference table provides baseline candidates, not the final answer, and must be verified with quantitative estimation.

Scenario	Baseline candidate	Key configuration	Verified reference
Dense small-to-medium scale	Pure TP	all tp = W	GPT-OSS 8-GPU
MoE Large-scale High-throughput	Attn DP + MoE EP + Differentiated TP	attn_tp=1, moe_tp=1	R1/V3.2/GLM-5/Kimi-K2 128-GPU
MoE Medium-scale	Hybrid EP+TP or Pure TP	Need to compare two options	LongCat 32-GPU / R1 16-GPU
MoE Small-scale Low-latency	Pure TP or attn_tp>1 + EP	attn_tp=W, moe_tp=1	To be verified
Long Sequence + MLA	Add CP	+cp_size	V3.2/GLM-5/Kimi-K2 64-GPU
Long Sequence + GQA	Large TP or Add CP	attn_tp=W or +cp_size	R1 32-GPU
Ultra-long Sequence	Add KVP	+kvp_size, oproj_tp=kvp_size	LongCat 131K
Advanced High-throughput	+AFD+EPLB+MTP	enable_afd, perfect_eplb	LongCat/R1 128-GPU

Verified Parallel Patterns

The repository contains the following verified module combination patterns (specific values see references/config-index.md）：

Pattern	Characteristics	Representative Models
Large-scale High-throughput (All MoE)	Attn DP + MoE EP, all tp=1	R1 128-GPU
Large-scale High-throughput (MLA + Shared Expert)	Attn DP + Dense TP + MoE EP + O_proj TP	V3.2/GLM-5/Kimi-K2 128-GPU
Large-scale High-throughput (AFD)	Same as above + Attention-FFN Separation	LongCat 128-GPU
Medium-scale Hybrid	Attn DP + Dense TP + MoE EP	LongCat 32-GPU
Small-scale Pure TP	Unified TP for all modules	R1/Qwen3 16-GPU
Long Sequence + CP (MLA)	Attn uses CP + MoE EP	V3.2/GLM-5/Kimi-K2 64-GPU
Long Sequence + Large TP (GQA)	Unified large TP for all non-MoE modules	R1 32-GPU

Usage: Select candidate patterns from the quick reference table, then use this table to confirm whether the pattern characteristics match. For quantitative estimation, refer to the reference to obtain specific values.

Completion Criteria

Candidate parallel patterns determined (one configuration + optional CP/KVP add-ons)
The TP degree direction for each module is determined, referencing verified configurations
Associated with the closest reference model in the repository

Step 3: Quantitative Estimation

Based on the candidate patterns, consult the reference configurations to obtain starting values, then perform quantitative evaluation to determine the specific values of parallel_config.

3.0 Consult Reference Configurations

From {file:./references/config-index.md} find the verified configuration that best matches the candidate pattern as the starting point for quantitative estimation:

Search by deployment scenario (Decode high-throughput/low-latency/hybrid, Prefill long sequence, ultra-long sequence)
Read the actual YAML configuration files and design documents of the matching model (matching table see the end of config-index)
Use verified values as the baseline, adjust based on parameter differences of the target model

3.1 Parameter Count Calculation

Standard GQA Attention：

Attention/层 = H × D_h × (2 × N_h + 2 × N_kv)    # Q, K, V, O_proj

MLA Attention(DeepSeek-V3/R1, LongCat-Flash, etc.):

Attention/层 ≈ H × q_lora_rank                      # q_down_proj
              + q_lora_rank × D_h × N_h             # q_up_proj
              + H × kv_lora_rank                     # kv_down_proj
              + kv_lora_rank × (N_kv × D_h × 2)     # kv_up_proj (K + V)
              + N_h × v_head_dim × H                 # o_proj

Note: The specific projection structure of MLA varies by model (whether there is rope separation, absorb mode, etc.), so this formula is approximate. For the exact parameter count, refer to the model config and the matrix dimensions in the code.

FFN / MoE / Embed：

FFN/层       = 3 × H × H_ffn                        # gate + up + down
MoE/层       = E × 3 × H × H_ffn                    # 所有专家
Embed        = V × H
LMHead       = V × H

P_total = (L - L_moe) × (P_attn + P_ffn)
        + L_moe × (P_attn + P_moe)
        + P_embed + P_lmhead

3.2 Single-card memory estimation

The following formula is rough sizing, used to determine whether a candidate strategy is memory-feasible. Exact values require actual runs or profiling data from verified configurations.

Parallelism derivation(Each module is independent; do not mix them):

attn_dp_size  = world_size // attn_tp_size
dense_dp_size = world_size // dense_tp_size
moe_ep_size   = world_size // moe_tp_size    # EP 度由 moe_tp_size 决定，与 attn_tp_size 无关

Parameter weight memory(Divide by the corresponding TP/EP degree for each module):

P_per_card = P_attn / attn_tp_size
           + P_ffn / dense_tp_size          # 非 MoE 层
           + P_moe / moe_ep_size            # MoE 层, moe_ep_size = world_size // moe_tp_size
           + P_embed / embed_tp_size
           + P_lmhead / lmhead_tp_size

参数显存 = P_per_card × bytes_per_param

bytes_per_param: currently only BF16 = 2 is supported

KV Cache memory(per card B_local = B / dp_size):

标准 GQA：B_local × S × L × 2 × N_kv × D_h × bytes_per_kv
MLA 压缩：B_local × S × L × kv_lora_rank × bytes_per_kv

Note: actual KV Cache is also affected by page/block allocation granularity, prefix cache, chunked prefill, etc.

Other memory overhead(The formula cannot estimate this precisely; pay attention to):

Quantization additional state (quantization is currently not supported; consider it in future extensions)
Communication buffer (AllReduce / AllToAll temporary buffers)
graph mode workspace
Activation temporary buffer (Decode is usually small; Prefill is strongly related to S, chunk size, and attention backend)

Feasibility judgment: parameter memory + KV Cache already close to card capacity × 0.8 → infeasible, need to increase parallelism

3.3 Communication analysis

The impact of communication on performance is not simply "large byte volume means slow". Analyze according to the following three layers, with priority from high to low.

Layer 1: Whether it is cross-node (largest impact)

Intra-node HCCS: ~56 GB/s (A2), low latency
Inter-node RDMA: bandwidth and latency are both far worse than intra-node
Judgment: tp_size > number of cards in a single node → cross-node, usually should be avoided

Layer 2: Communication primitive type

Primitive	Trigger scenario	Characteristics
AllReduce	RowParallel output of TP	Data volume = tensor_size × 2(tp-1)/tp
AllToAll	Token dispatch/combine of EP	Data volume depends on the number of activated experts and token distribution
AllGather / ReduceScatter	TP degree switching boundary between modules	Data volume = tensor_size
Send/Recv	KV shard exchange of CP, AFD communication	Point-to-point, can overlap with computation

Layer 3: Whether it can be overlapped by computation

Multi-stream parallelism can hide part of the communication latency (Send/Recv + Compute in parallel)
AllReduce / AllToAll are usually on the critical path and difficult to fully overlap
Repository practice: LongCat AFD uses Send/Recv instead of AllReduce precisely to exploit overlap

Priority order: whether communication is cross-node > primitive type > communication byte volume > whether it can overlap

3.4 Adjust strategy based on estimation results

After estimation is complete, check and adjust according to the following criteria:

Memory feasibility：

Total memory > card capacity × 0.95 → need to increase parallelism or reduce batch
KV Cache proportion > 50% of available memory → long-sequence scenario, consider KVP / CP / KVCache Offload / reducing moe_chunk
Repository practice: R1 long sequence moe_chunk=512, LongCat 131K uses KVP=8

Communication feasibility：

tp_size > number of GPUs per node → TP communication crosses nodes, performance drops sharply, must adjust
Intra-node HCCS: ~56 GB/s (A2), inter-node RDMA: far lower than this
Repository practice: TP communication for all models is kept within a node
Estimated communication time > computation time → TP degree too large, reduce TP or switch to EP
Repository practice: large-scale MoE Decode uses EP (AllToAll) instead of TP (AllReduce) precisely because of this

Batch throughput：

B_local = B / dp_size too small (e.g., < 16) → DP degree insufficient, throughput limited
Ways to increase DP: reduce TP degree (dp_size = world_size / tp_size)
Repository practice: 128-GPU Decode attn_tp=1 makes attn_dp=128, maximizing throughput

Prefill TTFT：

Single-GPU Prefill computation too large (S > 4K) → needs TP or CP to share the load
MLA models prefer CP, standard GQA prefers TP (see decision tree for reasons)

If estimation shows the reference values of a validated configuration are reasonable (memory feasible + communication within node), prefer directly adopting the validated values without re-deriving.

Completion criteria

Single-GPU memory for each candidate strategy has been calculated
Communication volume has been estimated
Strategies have been ranked (with quantitative basis)
Specific values of parallel_config have been determined

Step 4: Solution review

Before output, check the following constraints:

Constraint checks

[A] Hard constraints(configuration is invalid if not satisfied):

For each module world_size % *_tp_size == 0
num_attention_heads % attn_tp_size == 0
num_key_value_heads % attn_tp_size == 0(implementation that splits by KV head)
num_experts % ep_size == 0(implementation requiring uniform expert partitioning)

[B] Strong empirical checks(usually should be satisfied; if not, sufficient justification is required):

Single-GPU memory estimate ≤ actual memory × 0.95
tp_size ≤ number of GPUs per node (avoid cross-node TP)
In the Decode phase, attn_tp is not greater than the necessary value (avoid ineffective communication when S=1)

[C] This repository's implementation checks(depends on the specific framework/model implementation):

dp_size is determined by world_size // tp_size automatically derived (feature of this framework, see inference_config.py)
In KVP mode oproj_tp_size == kvp_size(constraint of this repository's KVP implementation)
N_h * v_head_dim % oproj_tp_size == 0(MLA o_proj splitting constraint)

Output format

The TP degree of each module is an independent decision, and candidate solutions should be fully combined (e.g., attn_tp=8 + moe_tp=1 + dense_tp=8); do not consider only the two extremes of "all TP" or "all EP". Output 2~3 candidate solutions and rank them, with each candidate explaining its trade-offs for high-throughput/low-latency/long-sequence scenarios:

# 候选 A（推荐）：{模式名称}
# 理由：{为什么推荐}
# 参考：{最接近的已验证配置}
candidate_a:
  parallel_config:
    attn_tp_size: {value}       # attn_dp_size = world_size // attn_tp_size
    dense_tp_size: {value}
    moe_tp_size: {value}        # moe_ep_size = world_size // moe_tp_size
    embed_tp_size: {value}
    lmhead_tp_size: {value}
    o_proj_tp_size: {value}     # MLA 模型需要，非 MLA 可省略
    cp_size: {value}            # 按需设置
    kvp_size: {value}           # 按需设置
  estimation:
    params_per_card_gb: {value}
    kv_cache_per_card_gb: {value}
    memory_feasible: true/false
    cross_node_comm: true/false
  tradeoff:
    throughput: {优势/中性/风险}
    latency: {优势/中性/风险}
    long_context: {优势/中性/风险}

# 候选 B（备选）：{模式名称}
# 理由：{适用场景 / 与 A 的权衡}
candidate_b:
  parallel_config: ...

# 注：dp_size/ep_size 不在 YAML 中配置，由框架自动推导

Ranking dimensions(by priority):

Memory feasibility (directly exclude if infeasible)
Cross-node communication risk (downgrade if present)
Expected performance (solutions showing structural advantages in quantitative estimation are prioritized)
Distance from validated configurations (when there is no performance difference, the closer the better)

When quantitative analysis clearly shows that a solution has a structural performance advantage (e.g., EP's complete matrix vs TP's fragmented matrix), even if the solution is not validated, it should be marked as "recommended for comparative testing" and not automatically downgraded due to distance from validated configurations.

Output the recommendation results in a structured format.

Weight handling tips

After configuration changes, note whether weights need to be reprocessed:

enable_online_split_weight: True: No re-conversion needed; runtime automatically splits
enable_online_split_weight: False: Must re-convert; weights are bound to parallel_config
For specific operations, see the model-infer-parallel-impl skill

Completion criteria

All hard constraints passed
Recommended configuration includes quantitative basis
Has been linked to repository reference configuration

(Optional) Step 5: Profiling calibration

When actual profiling data is available, calibrate the estimates from Step 3.

Three key metrics needed

Metric 1: Communication ratio(most important)

Extract from the trace the sum of all Hccl* operator durations / total single-step duration:

< 20%: strategy is reasonable
20-40%: room for optimization; check whether TP communication crosses nodes
> 40%: strategy has issues; adjust the TP/EP ratio

Where to view: TensorBoard Trace View, filter HcclAllReduce、HcclAllToAll and other operators

Metric 2: Peak memory

Obtain the peak for each Rank via torch.npu.max_memory_allocated() :

Verify the estimation deviation from Step 3
Determine how large batch_size can be set

Current repository profiler configuration profile_memory=False, requires inserting an additional line of code to obtain

Metric 3: Inter-Rank duration variance(MoE EP specific)

Compare the total single-step duration of each Rank, calculate (max - min) / mean:

< 5%: good balance
5-15%: consider enabling perfect_eplb: True
> 15%: the EP strategy may not suit the current workload

Where to view: independent trace files for each Rank ({res_path}/prof/ directory)

Deviation handling

Deviation range	Action
< 10%	Strategy confirmed, no adjustment needed
10-30%	Fine-tune the deviating module (e.g., adjust tp_size so communication stays within nodes)
> 30%	Re-evaluate the strategy; consider module-level differentiated parallelism

Common errors

Error pattern	Root cause	Prevention
Skip analysis and directly use default all-TP	Communication overhead not evaluated	Must complete communication volume estimation before deciding configuration
For small-scale MoE model deployment, only consider one option	Did not compare the communication efficiency of EP and TP	For small-to-medium scale, compare EP and TP; refer to the two configurations for R1 16 cards
Ignore the tp_size of Embed/LMHead	In large-vocabulary models, these two layers account for a non-negligible share of memory	When V > 100K, set independently; refer to V3.2/Kimi-K2 embed=16
Memory estimation omits KV Cache	During Decode with long sequences, KV Cache is the main memory consumer	KV Cache must be calculated separately; for MLA, estimate using kv_lora_rank
Apply TP to Attention in MLA models	MLA KV Cache is a compressed vector and is not stored by head; TP yields no benefit	For MLA models, attn_tp=1; use CP instead for Prefill
Only look at Decode or only at Prefill metrics on one side	In single-machine deployment, both stages share one parallel_config; optimizing one side may degrade overall performance	Evaluate throughput, TTFT, and long-sequence requirements simultaneously under a single configuration
TP communication crosses nodes	Cross-node RDMA bandwidth is far lower than intra-node HCCS	tp_size ≤ number of cards in a single node
Changed parallel_config but did not reconvert the weights	The weights and configuration of offline split are bound together	It is recommended to enable enable_online_split_weight, or rerun weight_convert after changing the configuration

For the complete repository reference configuration index, see {file:./references/config-index.md}

About this skill

Scope of Application

Analysis Process

Step 1: Extract Model Parameters and Module Links

1.1 Basic Parameters

1.2 Module Link Identification

1.3 Deployment Information Confirmation

Completion criteria

Step 2: Qualitative Classification

Unified decision tree

Layer 1: Architecture Backbone

Layer 2: Scenario Adjustment

Layer 3: Sequence Length Add-on

Quick reference table

Verified Parallel Patterns

Completion Criteria

Step 3: Quantitative Estimation

3.0 Consult Reference Configurations

3.1 Parameter Count Calculation

3.2 Single-card memory estimation

3.3 Communication analysis

3.4 Adjust strategy based on estimation results

Completion criteria

Step 4: Solution review

Constraint checks

Output format

Weight handling tips

Completion criteria

(Optional) Step 5: Profiling calibration

Three key metrics needed

Deviation handling

Common errors