SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillshccl-test

hccl-test

Testing & quality

View repository

Updated 10-15 23:45

About this skill

HCCL Performance Test

The HCCL performance test tool is used to test the functional correctness and performance of HCCL (Huawei Collective Communication Library) collective communication.

Overview

  • Applicable Scenarios: Collective communication performance testing in distributed training scenarios
  • Source Code Location：${INSTALL_DIR}/tools/hccl_test
  • Supported Versions：CANN 8.3.RC1, CANN 8.5, CANN 25.RC

Supported Product Models

Product Series Maximum Number of Ranks Remarks
Atlas Training Series Products 4096 -
Atlas A2 Training Series Products 32K -
Atlas A3 Training Series Products/Atlas A3 Inference Series Products 32K AlltoAll/AlltoAllV maximum 8K
Atlas 300I Duo Inference Card - -

Core Operators (Recommended for Testing)

Most commonly used in distributed training scenarios:

Operator Executable File Communication Mode Applicable Scenarios Recommendation Level
AllReduce all_reduce_test Many-to-Many Gradient aggregation, parameter synchronization ⭐⭐⭐ Must Test
AllGather all_gather_test Many-to-Many Data aggregation, parameter collection ⭐⭐⭐ Must Test
Broadcast broadcast_test One-to-Many Configuration distribution, initialization ⭐⭐ Optional
AlltoAll alltoall_test Many-to-Many Data rearrangement, load balancing ⭐⭐ Optional

Note: For the complete operator list, see references/parameters.md

Quick Reference

# 1. 前置检查（多机测试必需）
./scripts/pre-test-check.sh 175.99.1.2 175.99.1.3

# 2. 编译工具
cd ${INSTALL_DIR}/tools/hccl_test
make MPI_HOME=/usr/local/mpich ASCEND_DIR=${INSTALL_DIR}

# 3. 快速连通性测试（单机）
mpirun -n 8 ./bin/all_reduce_test -p 8 -b 8K -e 64M -f 2 -d fp32 -o sum

# 4. 完整性能测试（多机，推荐）
mpirun -f hostfile -n 16 ./bin/all_reduce_test -p 8 -b 8K -e 1G -f 2 -d fp32 -o sum

1. Pre-test Checklist (Required for Multi-Node Testing)

⚠️ Important: The following checks must be completed before multi-node testing; otherwise, link establishment timeouts or test failures may occur.

1.1 SSH Passwordless Login Configuration

SSH passwordless login must be configured between all nodes:

# 1. 生成 SSH 密钥（如已存在可跳过）
ssh-keygen -t rsa

# 2. 将公钥复制到所有节点（包括本机）
ssh-copy-id -i ~/.ssh/id_rsa.pub root@<node1_ip>
ssh-copy-id -i ~/.ssh/id_rsa.pub root@<node2_ip>

# 3. 验证免密登录
ssh root@<node1_ip> "echo 'SSH OK'"
ssh root@<node2_ip> "echo 'SSH OK'"

1.2 CANN Version Consistency Check

The CANN versions of multiple nodes must be consistent; otherwise, the test will fail:

# 检查所有节点的 CANN 版本
for node in 175.99.1.2 175.99.1.3; do
    echo "=== $node ==="
    ssh root@$node "cat /usr/local/Ascend/ascend-toolkit/latest/version.cfg | grep runtime_running_version"
done

Note: When versions are inconsistent, they need to be unified (it is recommended to unify to the latest RC version).

1.3 NPU Health Status Check

Before testing, confirm that all NPUs are in normal status:

# 检查所有节点的 NPU 健康状态
for node in 175.99.1.2 175.99.1.3; do
    echo "=== $node NPU Health ==="
    ssh root@$node "npu-smi info -t health -i 0"
done

NPU Status Description：

Status Description Recommended Action
OK Normal ✅ Can be used
Alarm Alarm ⚠️ Troubleshooting required
Offline Offline ❌ Cannot be used

If an Alarm status exists, the faulty card needs to be excluded. For example, if NPU 0 is faulty, use 7 cards for testing.

1.4 One-Click Pre-Check Script

# 使用提供的检查脚本（推荐）
./scripts/pre-test-check.sh 175.99.1.2 175.99.1.3

2. MPI Installation

The HCCL performance test tool relies on MPI to launch multiple processes, and by default uses MPICH。

2.1 MPICH Installation (Recommended)

Download Address：https://www.mpich.org/static/downloads/

Product Series Recommended Version
Atlas A3 Training Series Products/Atlas A3 Inference Series Products MPICH 4.1.3
Atlas A2 Training Series Products MPICH 3.2.1
Atlas Training Series Products MPICH 3.2.1
Atlas 300I Duo Inference Card MPICH 3.2.1

Installation Steps：

# 1. 解压
tar -zxvf mpich-${version}.tar.gz
cd mpich-${version}

# 2. 配置编译选项
# Atlas A3 产品（必须使用 TCP 协议）
./configure --disable-fortran --prefix=/usr/local/mpich --with-device=ch3:nemesis

# 其他产品
./configure --disable-fortran --prefix=/usr/local/mpich

# 3. 编译安装
make -j 32 && make install

2.2 Open MPI Installation (Alternative)

Applicable to scenarios that require IPv6 support.

tar -zxvf openmpi-4.1.5.tar.gz
cd openmpi-4.1.5
./configure --disable-fortran --enable-ipv6 --prefix=/usr/local/openmpi
make -j 32 && make install

2.3 Environment Configuration

# MPICH 环境
export INSTALL_DIR=/usr/local/Ascend/ascend-toolkit/latest
export PATH=/usr/local/mpich/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/mpich/lib:${INSTALL_DIR}/lib64:$LD_LIBRARY_PATH

3. Tool Compilation

cd ${INSTALL_DIR}/tools/hccl_test

# MPICH
make MPI_HOME=/usr/local/mpich ASCEND_DIR=${INSTALL_DIR}

# Open MPI
make MPI_HOME=/usr/local/openmpi ASCEND_DIR=${INSTALL_DIR}

After successful compilation, in the bin directory, 10 executable files are generated.

4. Testing Scenarios

4.1 Quick Connectivity Test

Used to verify basic HCCL connectivity; the data volume is small, and execution is fast:

# 单机快速测试
mpirun -n 8 ./bin/all_reduce_test -p 8 -b 8K -e 64M -f 2 -d fp32 -o sum

# 多机快速测试
mpirun -f hostfile -n 16 ./bin/all_reduce_test -p 8 -b 8K -e 64M -f 2 -d fp32 -o sum

4.2 Full Performance Test (Recommended)

Used to test large-bandwidth network performance, with data volume up to 1GB, which better reflects actual training scenarios:

# 单机完整性能测试
mpirun -n 8 ./bin/all_reduce_test -p 8 -b 8K -e 1G -f 2 -d fp32 -o sum

# 多机完整性能测试（推荐）
mpirun -f hostfile -n 16 ./bin/all_reduce_test -p 8 -b 8K -e 1G -f 2 -d fp32 -o sum

Parameter Description：

  • -b 8K: Starting data volume 8KB
  • -e 1G: Ending data volume 1GB (64M can only test small data volumes)
  • -f 2: Multiplication factor, tests 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1M, 2M, 4M, 8M, 16M, 32M, 64M, 128M, 256M, 512M, 1G

4.3 Core Operator Test

# 使用 quick-verify.sh 测试核心算子（推荐）
./scripts/quick-verify.sh 8

# 手动测试核心算子
./bin/all_reduce_test -p 8 -b 8K -e 1G -f 2 -d fp32 -o sum
./bin/all_gather_test -p 8 -b 8K -e 1G -f 2 -d fp32

4.4 Hostfile Configuration

MPICH Format (节点IP:卡数)：

# 单机测试
175.99.1.3:8

# 双机测试
175.99.1.3:8
175.99.1.4:8

Please configure the AI Server information belonging to the same supernode together; cross-configuration is not supported.

5. Parameters

5.1 Core Parameter Quick Reference

Parameter Description Example
-p <npus> Number of NPUs participating in training on a single node -p 8
-b <size> Starting value of test data volume -b 8K
-e <size> Ending value of test data volume -e 1G
-f <factor> Multiplication factor -f 2
-d <type> Data type: fp32/fp16/int32 -d fp32
-o <op> Operation type: sum/prod/max/min -o sum
-n <iters> Number of iterations (default 20) -n 20
-c <0/1> Whether to enable result verification (default 1) -c 1

For detailed parameter descriptions, see references/parameters.md

5.2 Data Volume Configuration Example

# 固定数据量测试
-b 100M -e 100M

# 乘法因子测试（测试 100M, 200M, 400M）
-b 100M -e 400M -f 2

# 完整性能测试（推荐）
-b 8K -e 1G -f 2

6. Results

6.1 Output Format

data_size      avg_time(us)    alg_bandwidth(GB/s)    check_result
8192           125.3           0.065                  success
16384          132.1           0.124                  success
...
Field Description
data_size Data volume participating in collective communication on a single NPU (Bytes)
avg_time Execution time of the collective communication operator (us)
alg_bandwidth Algorithm bandwidth (GB/s), calculation method: collective communication data volume/time taken
check_result Result verification identifier: success/failed/NULL

6.2 Result Parsing

# 解析结果文件
./scripts/parse-hccl-result.py output.log

# 输出 Markdown 表格
./scripts/parse-hccl-result.py output.log -f markdown

7. Actual Test Results

7.1 Dual-Machine 16×910B3 Test Data

We tested all 10 operators in a dual-machine 16×910B3 environment:

Operator Result Remarks
AllReduce ✅ PASS Maximum bandwidth 48.59 GB/s (32MB)
AllGather ✅ PASS -
AllGatherV ❌ FAIL retcode: 5 (variable-length parameter issue)
AlltoAll ✅ PASS -
AlltoAllV ✅ PASS -
Broadcast ✅ PASS -
Reduce ✅ PASS -
ReduceScatter ✅ PASS -
ReduceScatterV ❌ FAIL retcode: 5 (variable-length parameter issue)
Scatter ✅ PASS -

Pass rate: 8/10 (80%), all core operators passed.

7.2 Test Environment Reference

  • Test node：175.100.2.3, 175.100.2.4
  • NPU: 910B3 × 8 per node (16 cards in total)
  • CANN：25.3.rc1
  • Communication NIC：enp189s0f0
  • MPI：MPICH 3.2.1

8. Common Issues

8.1 Prerequisite Issues

Issue Cause Solution
SSH passwordless login failed SSH key not configured Run ssh-copy-id to configure passwordless login
Inconsistent CANN versions Different CANN versions across multiple machines Unify the CANN version across all nodes
NPU Alarm status NPU hardware failure Check npu-smi info -t health to identify faulty cards

8.2 Runtime Issues

Issue Cause Solution
gethostbyname failed Hostname cannot be resolved Configure /etc/hosts
retcode: 7 Residual process interference Run cleanup command:mpirun -f hostfile -n 16 pkill -9 -f "all_reduce_test"
retcode: 5 Variable-length parameter configuration error AllGatherV/ReduceScatterV require special parameter configuration

8.3 Other Notes

  1. Docker container: If using a Docker container for testing, the host network mode is required
  2. Log Viewing: When a test fails, you can view the latest logs in ~/ascend/log/debug/plog latest logs in
  3. Process Cleanup: Before testing, check whether other processes are occupying the card, and clean them up manually if necessary

For detailed troubleshooting, see references/common-issues.md

9. Scripts

9.1 Pre-check Script

# 多机测试前置检查
./scripts/pre-test-check.sh 175.99.1.2 175.99.1.3

Check items:

  • SSH passwordless login
  • CANN version consistency
  • NPU health status
  • Network connectivity

9.2 Quick Verification Script

# 测试核心算子
./scripts/quick-verify.sh 8

# 完整性能测试
./scripts/quick-verify.sh 8 full

9.3 Multi-machine Test Script

# 一键多机测试（自动前置检查 + 测试）
./scripts/multi-node-test.sh --nodes 175.99.1.2,175.99.1.3 --npus 8 --mode full

Official References