training-mfu-calculator

MFU Calculation Tool

When to Use This Skill

• User needs tocalculate MFU: evaluate hardware utilization for large model training.
• User mentionsFLOPs, throughput, training efficiency: analyze model training performance.
• User needsperformance evaluation: compare training efficiency across different hardware or configurations.
• User mentionshardware utilization, compute utilization: optimize training system performance.
• User needsmodel performance analysis: analyze the computation amount of Dense or MoE models.

---

Quick Start

The simplest way to use it

from mfu_calculator import MODEL_CONFIGS, MFUCalculator, TrainingConfig

Use predefined model configurations

model_config = MODEL_CONFIGS["llama-7b"]

Training configuration

training_config = TrainingConfig( batch_size=512, micro_batch_size=4, num_gpus=128, seq_length=2048, step_time=2.5, # time per step (seconds) hardware_peak_flops=312, # A100 peak compute hardware_name="A100", )

Calculate MFU

calculator = MFUCalculator(model_config, training_config) print(calculator.generate_report())

---

## 信息来源

计算MFU需要三类信息，以下是获取方式：

### 1. 模型配置（从 config.json 读取）

从 HuggingFace 格式的 `config.json` 文件中读取模型架构参数：

```json
{
  "hidden_size": 4096,
  "num_hidden_layers": 94,
  "vocab_size": 151936,
  "num_attention_heads": 64,
  "num_key_value_heads": 4,
  "head_dim": 128,
  "intermediate_size": 12288,
  "moe_intermediate_size": 1536,
  "num_experts_per_tok": 8,
  "num_experts": 128
}

Parameter Mapping Table：

config.json Field	MFU Calculation Parameter	Description
hidden_size	hidden_size	Hidden layer dimension
num_hidden_layers	layer_num	Number of layers
vocab_size	vocab_size	Vocabulary size
num_attention_heads	head_num	Number of attention heads
num_key_value_heads	kv_head_num	Number of KV heads (GQA)
head_dim	head_dim	Head dimension
intermediate_size	intermediate_size	FFN intermediate layer
moe_intermediate_size	expert_hidden_size	MoE expert intermediate layer
num_experts_per_tok	topk	Number of activated experts

2. Training configuration (read from the launch script)

Read from the training launch script (e.g., run_train.sh、train.py parameters):

Common launch script examples：

# 示例1: torchrun 启动
torchrun --nproc_per_node=8 --nnodes=4 train.py \
    --seq_length 8192 \
    --global_batch_size 32 \
    ...

# 示例2: accelerate 启动
accelerate launch --num_processes 32 train.py \
    --max_seq_length 8192 \
    --train_batch_size 32 \
    ...

Parameter Mapping Table：

Launch Script Parameter	MFU Calculation Parameter	Description
--seq_length / --max_seq_length	seq_length	Training sequence length
--global_batch_size / --train_batch_size	gbs	Global batch size
--nproc_per_node × --nnodes / --num_processes	num_gpus	Total number of GPUs/NPUs

Calculate the number of GPUs/NPUs：

# 单机多卡
num_gpus = nproc_per_node

# 多机多卡
num_gpus = nproc_per_node * nnodes

3. Performance data (provided by the user or training logs)

Method 1: Provided by the user

The user directly provides the time per step (step_time):

• Unit: seconds
• Suggestion: take the average over a stable training phase

Method 2: Read from training logs

Extract step_time from training logs:

Log example 1 (Transformers format)：

[2024-01-15 10:23:45] step=100 loss=2.345 learning_rate=1e-4 step_time=4.4s
[2024-01-15 10:23:50] step=101 loss=2.342 learning_rate=1e-4 step_time=4.3s

Log example 2 (Megatron format)：

iteration 100/ 1000 | elapsed time per iteration (ms): 4400 | ...
iteration 101/ 1000 | elapsed time per iteration (ms): 4350 | ...

Log example 3 (PyTorch format)：

Step 100: loss=2.345, time=4.40s, throughput=1861.82 tokens/s/GPU

Extraction method：

import re

def extract_step_time(log_file):
    """从日志文件提取step_time"""
    times = []
    with open(log_file, 'r') as f:
        for line in f:
            # 匹配 "step_time=4.4s" 或 "time=4.40s"
            match = re.search(r'(?:step_)?time[=:]\s*([\d.]+)\s*s?', line)
            if match:
                times.append(float(match.group(1)))
    
    # 取稳定阶段的平均值（跳过前10步预热）
    return sum(times[10:]) / len(times[10:]) if len(times) > 10 else None

4. Hardware information

Default value：Ascend910B2 = 353 TFLOPS

How to obtain：

• From environment variables:echo $ASCEND_DEVICE_TYPE
• From npu-smi:npu-smi info
• From nvidia-smi:nvidia-smi --query-gpu=name --format=csv

---

Directory structure

agent_skills/mfu-calculator/
├── SKILL.md                          # 本文件
├── reference/
│   └── mfu_reference.md              # MFU计算参考文档
└── scripts/
    └── mfu_calculator.py             # MFU计算工具实现

---

Core Features

1. Supported model types

Dense models：

• ✅ LLaMA series (7B, 13B, 70B)
• ✅ Qwen series (7B, 72B)
• ✅ Custom Dense models

MoE models：

• ✅ Mixtral-8x7B
• ✅ Custom MoE models

2. Supported FFN Types

• ✅ Standard FFN
• ✅ SwiGLU FFN

3. Supported Attention Types

• ✅ Standard Multi-Head Attention
• ✅ Grouped Query Attention (GQA)

---

Usage

Method 1: Using Predefined Models

from mfu_calculator import MODEL_CONFIGS, MFUCalculator, TrainingConfig

# 查看支持的预定义模型
print(MODEL_CONFIGS.keys())
# 输出: dict_keys(['llama-7b', 'llama-13b', 'llama-70b', 'qwen-7b', 'qwen-72b', 'mixtral-8x7b'])

# 选择模型
model_config = MODEL_CONFIGS["llama-70b"]

# 配置训练参数
training_config = TrainingConfig(
    batch_size=1024,
    micro_batch_size=8,
    num_gpus=256,
    seq_length=2048,
    step_time=3.8,
    hardware_peak_flops=989,
    hardware_name="H100",
)

# 计算MFU
calculator = MFUCalculator(model_config, training_config)
print(calculator.generate_report())

Method 2: Custom Models

from mfu_calculator import ModelConfig, MFUCalculator, TrainingConfig

# 自定义Dense模型
model_config = ModelConfig(
    hidden_size=8192,
    num_layers=80,
    vocab_size=128000,
    seq_length=4096,
    num_attention_heads=64,
    num_key_value_heads=8,  # GQA
    intermediate_size=22016,
    ffn_type="swiglu",
)

# 训练配置
training_config = TrainingConfig(
    batch_size=2048,
    micro_batch_size=16,
    num_gpus=512,
    seq_length=4096,
    step_time=5.2,
    hardware_peak_flops=313,
    hardware_name="Ascend910B",
)

# 计算MFU
calculator = MFUCalculator(model_config, training_config)
mfu = calculator.calculate_mfu()
print(f"MFU: {mfu*100:.2f}%")

Method 3: MoE Models

from mfu_calculator import ModelConfig, MFUCalculator, TrainingConfig

# MoE模型配置
model_config = ModelConfig(
    hidden_size=4096,
    num_layers=32,
    vocab_size=32000,
    seq_length=2048,
    num_attention_heads=32,
    intermediate_size=14336,
    ffn_type="swiglu",
    is_moe=True,
    num_experts=8,
    num_experts_per_tok=2,
    expert_intermediate_size=14336,
)

# 训练配置
training_config = TrainingConfig(
    batch_size=512,
    num_gpus=128,
    seq_length=2048,
    step_time=4.1,
    hardware_peak_flops=312,
    hardware_name="A100",
)

# 计算MFU
calculator = MFUCalculator(model_config, training_config)
print(calculator.generate_report())

Method 4: Automatically Reading from config.json

import json
from mfu_calculator import ModelConfig, MFUCalculator, TrainingConfig, cal_flops_simple, cal_mfu_simple

# 1. 从 config.json 读取模型配置
with open("config.json", "r") as f:
    config = json.load(f)

# 2. 从启动脚本或用户提供训练配置
seq_length = 8192      # 从 --seq_length 参数
gbs = 32               # 从 --global_batch_size 参数
num_gpus = 32          # 从 nproc_per_node * nnodes 计算
step_time = 4.4        # 从训练日志或用户提供

# 3. 计算MFU
flops = cal_flops_simple(
    hidden_size=config["hidden_size"],
    expert_hidden_size=config.get("moe_intermediate_size", config.get("intermediate_size", 4 * config["hidden_size"])),
    head_num=config["num_attention_heads"],
    kv_head_num=config.get("num_key_value_heads", config["num_attention_heads"]),
    sequence_length=seq_length,
    layer_num=config["num_hidden_layers"],
    vocab_size=config["vocab_size"],
    topk=config.get("num_experts_per_tok", 1),
    gbs=gbs,
    head_dim=config.get("head_dim", config["hidden_size"] // config["num_attention_heads"])
)

# 默认使用 Ascend910B2 的峰值算力
hw_flops = 353 * 1e12

mfu = cal_mfu_simple(
    real_flops=flops,
    num_gpu=num_gpus,
    sec_per_step=step_time,
    hw_flops_per_gpu=hw_flops
)

print(f"模型总FLOPs: {flops/1e15:.2f} PFLOPs")
print(f"MFU: {mfu * 100:.2f}%")

Method 5: Complete Qwen3 MoE Example

import json
from mfu_calculator import cal_flops_simple, cal_mfu_simple

# Qwen3 MoE config.json
config = {
    "hidden_size": 4096,
    "num_hidden_layers": 94,
    "vocab_size": 151936,
    "num_attention_heads": 64,
    "num_key_value_heads": 4,
    "head_dim": 128,
    "intermediate_size": 12288,
    "moe_intermediate_size": 1536,
    "num_experts_per_tok": 8,
    "num_experts": 128
}

# 训练配置（从启动脚本读取）
seq_length = 8192
gbs = 32
num_gpus = 32
step_time = 4.4  # 用户提供或从日志读取

# 计算FLOPs
flops = cal_flops_simple(
    hidden_size=config["hidden_size"],
    expert_hidden_size=config["moe_intermediate_size"],
    head_num=config["num_attention_heads"],
    kv_head_num=config["num_key_value_heads"],
    sequence_length=seq_length,
    layer_num=config["num_hidden_layers"],
    vocab_size=config["vocab_size"],
    topk=config["num_experts_per_tok"],
    gbs=gbs,
    head_dim=config["head_dim"]
)

# 计算MFU（默认 Ascend910B2）
hw_flops = 353 * 1e12
mfu = cal_mfu_simple(
    real_flops=flops,
    num_gpu=num_gpus,
    sec_per_step=step_time,
    hw_flops_per_gpu=hw_flops
)

print(f"模型总FLOPs: {flops/1e15:.2f} PFLOPs")
print(f"MFU: {mfu * 100:.2f}%")
# 输出:
# 模型总FLOPs: 9.83 PFLOPs
# MFU: 19.78%

---

Performance Metrics

1. MFU (Model FLOPs Utilization)

Calculation Formula：

MFU = 有效模型计算FLOPs / 硬件理论峰值FLOPs

Evaluation Criteria：

MFU Range	Evaluation Level	Description
≥40%	Excellent	Very high hardware utilization
30-40%	Good	Relatively good hardware utilization
20-30%	Average	Room for optimization
<20%	Needs optimization	Obvious performance issues exist

2. Single-Card Throughput

Calculation Formula：

Throughput = (gbs × seq_length) / (step_time × num_gpus)

Unit: tokens/s/GPU

3. Cluster Throughput

Calculation Formula：

Cluster_Throughput = (Throughput × num_gpus × 3600 × 24) / 10^12

Unit: T tokens/day

---

Hardware Peak FLOPS Reference

NVIDIA GPU

GPU Model	FP16/BF16 TFLOPS
A100	312
H100	989
V100	125
RTX4090	165

Ascend NPU

NPU Model	FP16 TFLOPS	Description
Ascend910	256	TSMC 7nm EUV
Ascend910A	256	TSMC 7nm EUV
Ascend910A2	256	TSMC 7nm EUV
Ascend910B	320	SMIC N+1
Ascend910B1	320	SMIC N+1
Ascend910B2	353	Default value
Ascend910B3	353	SMIC N+1
Ascend910C	800	Dual-die package

---

FLOPs Calculation Formula

Attention Part

Standard Attention：

FLOPs = 8BSH² + 4BS²H

GQA (Grouped Query Attention)：

FLOPs = BSH² × (4 + kv_heads/num_heads) + 4BS²H

FFN Part

Standard FFN：

FLOPs = 4BSH × intermediate_size

SwiGLU FFN：

FLOPs = 6BSH × intermediate_size

MoE + SwiGLU：

FLOPs = 6BSH × activated_experts × expert_intermediate_size

Logits Part

FLOPs = 6BSH × vocab_size

Total FLOPs

单层FLOPs = attention_flops + ffn_flops
模型FLOPs = 单层FLOPs × num_layers
单步FLOPs = 3 × 模型FLOPs + logits_flops

Description: Assume backward pass FLOPs = 2 × forward pass FLOPs

---

Output Example

============================================================
MFU计算报告
============================================================

【模型配置】
- 模型类型: Dense
- 隐藏层维度: 4096
- 层数: 32
- 注意力头数: 32
- KV头数: 32
- 序列长度: 2048
- 词表大小: 32000
- FFN类型: swiglu
- FFN中间层大小: 11008

【训练配置】
- 全局batch size: 512
- 微批次大小: 4
- GPU数量: 128
- 每步时间: 2.500秒
- 硬件: A100
- 硬件峰值: 312 TFLOPS

【FLOPs分析】
- 单步训练FLOPs: 1.23e+18
- 有效计算FLOPS: 4.92e+17 (492.00 TFLOPS)

【性能指标】
- MFU: 52.31%
- 单卡吞吐: 3276.80 tokens/s/GPU
- 集群吞吐: 36.28 T tokens/day

【性能评估】
- MFU评估: 良好 (50-60%)
============================================================

---

FAQ

Q1: What should I do if MFU is too low?

Possible Causes：

• Excessive communication overhead
• Memory access latency
• Data loading bottleneck
• CPU-GPU synchronization

Solutions：

• Optimize communication efficiency (use NCCL/HCCL optimization)
• Reduce memory fragmentation
• Optimize the data loading pipeline
• Use mixed precision training

Q2: How do I choose an appropriate hardware peak FLOPS?

Recommendations：

• Use the FP16/BF16 peak compute power from the hardware specifications
• Consider the data type actually used
• Refer to the hardware peak FLOPS table in this document

Q3: What is special about MFU calculation for MoE models?

Description：

• MoE models only count activated experts
• Actual computation = base computation × number of activated experts
• Need to configure correctlynum_experts_per_tokparameters

Q4: What impact does GQA have on FLOPs?

Description：

• GQA reduces KV cache and computation
• The calculation formula needs to considerkv_heads/num_headsthe ratio
• It can significantly reduce the computational overhead of long sequences

---

Best Practices

1. Accurately measure step_time

import time

# 测量多个步骤取平均值
step_times = []
for i in range(10):
    start = time.time()
    train_step()
    step_times.append(time.time() - start)

avg_step_time = sum(step_times) / len(step_times)

2. Configure batch_size reasonably

# 根据显存和硬件选择合适的micro_batch_size
# 一般建议：尽可能大，但不要OOM

# A100 80GB示例
micro_batch_size = 8  # 对于7B模型

# H100 80GB示例
micro_batch_size = 16  # 对于7B模型

3. Compare different configurations

# 对比不同硬件的MFU
configs = [
    ("A100", 312),
    ("H100", 989),
    ("Ascend910B", 313),
]

for name, peak_flops in configs:
    training_config.hardware_name = name
    training_config.hardware_peak_flops = peak_flops
    calculator = MFUCalculator(model_config, training_config)
    mfu = calculator.calculate_mfu()
    print(f"{name}: MFU = {mfu*100:.2f}%")

---

Reference Resources

Documentation

• MFU Calculation Reference: reference/mfu_reference.md
• MFU Calculation Tool: scripts/mfu_calculator.py

External Resources

• Large Language Model Training - LLM: Dense & MOE Model MFU Calculation
• PyTorch FLOPs Calculation
• Transformer Paper

---

Summary

This skill provides a complete MFU calculation solution:

1. Comprehensive Support: Supports Dense and MoE models, multiple FFN and Attention types
2. Easy to Use: Provides predefined model configurations for quick start
3. Detailed Report: Generates a complete performance analysis report
4. Accurate Calculation: Based on standard FLOPs calculation formulas

With this skill, you can accurately evaluate the hardware utilization of large model training and guide performance optimization work.