SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillshccl-test

hccl-test

Testing & quality

View repository

Updated 10-16 05:00

About this skill

HCCL Performance Test

The HCCL performance test tool is used to test the functional correctness and performance of HCCL (Huawei Collective Communication Library) collective communication.

Overview

  • Applicable Scenarios: Collective communication performance testing in distributed training scenarios
  • Source Code Location：${INSTALL_DIR}/tools/hccl_test
  • Supported Versions：CANN 8.3.RC1, CANN 8.5, CANN 25.RC

Supported Product Models

Product Series Maximum Number of Ranks Remarks
Atlas Training Series Products 4096 -
Atlas A2 Training Series Products 32K -
Atlas A3 Training Series Products/Atlas A3 Inference Series Products 32K AlltoAll/AlltoAllV maximum 8K
Atlas 300I Duo Inference Card - -

Core Operators (Recommended for Testing)

Most commonly used in distributed training scenarios:

Operator Executable File Communication Mode Applicable Scenario Recommendation Level
AllReduce all_reduce_test Many-to-Many Gradient aggregation, parameter synchronization ⭐⭐⭐ Must Test
AllGather all_gather_test Many-to-Many Data aggregation, parameter collection ⭐⭐⭐ Must Test
Broadcast broadcast_test One-to-Many Configuration distribution, initialization ⭐⭐ Optional
AlltoAll alltoall_test Many-to-Many Data rearrangement, load balancing ⭐⭐ Optional

Note: For the complete operator list, see references/parameters.md

Quick Reference

# 1. 前置检查（多机测试必需）
./scripts/pre-test-check.sh 175.99.1.2 175.99.1.3

# 2. 编译工具
cd ${INSTALL_DIR}/tools/hccl_test
make MPI_HOME=/usr/local/mpich ASCEND_DIR=${INSTALL_DIR}

# 3. 快速连通性测试（单机）
mpirun -n 8 ./bin/all_reduce_test -p 8 -b 8K -e 64M -f 2 -d fp32 -o sum

# 4. 完整性能测试（多机，推荐）
mpirun -f hostfile -n 16 ./bin/all_reduce_test -p 8 -b 8K -e 1G -f 2 -d fp32 -o sum

1. Pre-test Checklist (Required for Multi-Node Testing)

⚠️ Important: The following checks must be completed before multi-node testing; otherwise, link establishment timeouts or test failures may occur.

1.1 SSH Passwordless Login Configuration

SSH passwordless login must be configured between all nodes:

# 1. 生成 SSH 密钥（如已存在可跳过）
ssh-keygen -t rsa

# 2. 将公钥复制到所有节点（包括本机）
ssh-copy-id -i ~/.ssh/id_rsa.pub root@<node1_ip>
ssh-copy-id -i ~/.ssh/id_rsa.pub root@<node2_ip>

# 3. 验证免密登录
ssh root@<node1_ip> "echo 'SSH OK'"
ssh root@<node2_ip> "echo 'SSH OK'"

1.2 CANN Version Consistency Check

CANN versions across multiple nodes must be consistent; otherwise, the test will fail:

# 检查所有节点的 CANN 版本
for node in 175.99.1.2 175.99.1.3; do
    echo "=== $node ==="
    ssh root@$node "cat /usr/local/Ascend/ascend-toolkit/latest/version.cfg | grep runtime_running_version"
done

Note: If versions are inconsistent, unify them (it is recommended to unify to the latest RC version).

1.3 NPU Health Status Check

Before testing, confirm that all NPUs are in normal status:

# 检查所有节点的 NPU 健康状态
for node in 175.99.1.2 175.99.1.3; do
    echo "=== $node NPU Health ==="
    ssh root@$node "npu-smi info -t health -i 0"
done

NPU Status Description：

Status Description Recommended Action
OK Normal ✅ Can be used
Alarm Alarm ⚠️ Troubleshooting required
Offline Offline ❌ Cannot be used

If an Alarm status exists, the faulty card must be excluded. For example, if NPU 0 is faulty, use 7 cards for testing.

1.4 One-Click Pre-Check Script

# 使用提供的检查脚本（推荐）
./scripts/pre-test-check.sh 175.99.1.2 175.99.1.3

2. MPI Installation

The HCCL performance test tool relies on MPI to launch multiple processes, and by default uses MPICH。

2.1 MPICH Installation (Recommended)

Download Address：https://www.mpich.org/static/downloads/

Product Series Recommended Version
Atlas A3 Training Series Products/Atlas A3 Inference Series Products MPICH 4.1.3
Atlas A2 Training Series Products MPICH 3.2.1
Atlas Training Series Products MPICH 3.2.1
Atlas 300I Duo Inference Card MPICH 3.2.1

Installation Steps：

# 1. 解压
tar -zxvf mpich-${version}.tar.gz
cd mpich-${version}

# 2. 配置编译选项
# Atlas A3 产品（必须使用 TCP 协议）
./configure --disable-fortran --prefix=/usr/local/mpich --with-device=ch3:nemesis

# 其他产品
./configure --disable-fortran --prefix=/usr/local/mpich

# 3. 编译安装
make -j 32 && make install

2.2 Open MPI Installation (Alternative)

Applicable to scenarios that require IPv6 support.

tar -zxvf openmpi-4.1.5.tar.gz
cd openmpi-4.1.5
./configure --disable-fortran --enable-ipv6 --prefix=/usr/local/openmpi
make -j 32 && make install

2.3 Environment Configuration

# MPICH 环境
export INSTALL_DIR=/usr/local/Ascend/ascend-toolkit/latest
export PATH=/usr/local/mpich/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/mpich/lib:${INSTALL_DIR}/lib64:$LD_LIBRARY_PATH

3. Tool Compilation

cd ${INSTALL_DIR}/tools/hccl_test

# MPICH
make MPI_HOME=/usr/local/mpich ASCEND_DIR=${INSTALL_DIR}

# Open MPI
make MPI_HOME=/usr/local/openmpi ASCEND_DIR=${INSTALL_DIR}

After successful compilation, in the bin directory, 10 executable files are generated.

4. Testing Scenarios

4.1 Quick Connectivity Test

Used to verify basic HCCL connectivity; the data volume is small and execution is fast:

# 单机快速测试
mpirun -n 8 ./bin/all_reduce_test -p 8 -b 8K -e 64M -f 2 -d fp32 -o sum

# 多机快速测试
mpirun -f hostfile -n 16 ./bin/all_reduce_test -p 8 -b 8K -e 64M -f 2 -d fp32 -o sum

4.2 Complete Performance Test (Recommended)

Used to test large-bandwidth network performance, with data volume up to 1GB, which better reflects actual training scenarios:

# 单机完整性能测试
mpirun -n 8 ./bin/all_reduce_test -p 8 -b 8K -e 1G -f 2 -d fp32 -o sum

# 多机完整性能测试（推荐）
mpirun -f hostfile -n 16 ./bin/all_reduce_test -p 8 -b 8K -e 1G -f 2 -d fp32 -o sum

Parameter Description：

  • -b 8K: Starting data volume 8KB
  • -e 1G: Ending data volume 1GB (64M can only test small data volumes)
  • -f 2: Multiplication factor, test 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1M, 2M, 4M, 8M, 16M, 32M, 64M, 128M, 256M, 512M, 1G

4.3 Core Operator Test

# 使用 quick-verify.sh 测试核心算子（推荐）
./scripts/quick-verify.sh 8

# 手动测试核心算子
./bin/all_reduce_test -p 8 -b 8K -e 1G -f 2 -d fp32 -o sum
./bin/all_gather_test -p 8 -b 8K -e 1G -f 2 -d fp32

4.4 Hostfile Configuration

MPICH Format (节点IP:卡数)：

# 单机测试
175.99.1.3:8

# 双机测试
175.99.1.3:8
175.99.1.4:8

Please configure the AI Server information belonging to the same supernode together; cross-configuration is not supported.

5. Parameters

5.1 Core Parameters Quick Reference

Parameter Description Example
-p <npus> Number of NPUs participating in training on a single node -p 8
-b <size> Starting value of test data volume -b 8K
-e <size> Ending value of test data volume -e 1G
-f <factor> Multiplication factor -f 2
-d <type> Data type: fp32/fp16/int32 -d fp32
-o <op> Operation type: sum/prod/max/min -o sum
-n <iters> Number of iterations (default 20) -n 20
-c <0/1> Whether to enable result verification (default 1) -c 1

See detailed parameter description in references/parameters.md

5.2 Data Volume Configuration Example

# 固定数据量测试
-b 100M -e 100M

# 乘法因子测试（测试 100M, 200M, 400M）
-b 100M -e 400M -f 2

# 完整性能测试（推荐）
-b 8K -e 1G -f 2

6. Results

6.1 Output Format

data_size      avg_time(us)    alg_bandwidth(GB/s)    check_result
8192           125.3           0.065                  success
16384          132.1           0.124                  success
...
Field Description
data_size Data volume participating in collective communication on a single NPU (Bytes)
avg_time Execution time of the collective communication operator (us)
alg_bandwidth Algorithm bandwidth (GB/s), calculation method: collective communication data volume/time consumed
check_result Result verification flag: success/failed/NULL

6.2 Result Parsing

# 解析结果文件
./scripts/parse-hccl-result.py output.log

# 输出 Markdown 表格
./scripts/parse-hccl-result.py output.log -f markdown

7. Actual Test Results

7.1 Dual-Machine 16×910B3 Test Data

We tested all 10 operators in a dual-machine 16×910B3 environment:

Operator Result Remarks
AllReduce ✅ PASS Highest bandwidth 48.59 GB/s (32MB)
AllGather ✅ PASS -
AllGatherV ❌ FAIL retcode: 5 (variable-length parameter issue)
AlltoAll ✅ PASS -
AlltoAllV ✅ PASS -
Broadcast ✅ PASS -
Reduce ✅ PASS -
ReduceScatter ✅ PASS -
ReduceScatterV ❌ FAIL retcode: 5 (variable-length parameter issue)
Scatter ✅ PASS -

Pass rate: 8/10 (80%), all core operators passed.

7.2 Test Environment Reference

  • Test node：175.100.2.3, 175.100.2.4
  • NPU: 910B3 × 8 per node (16 cards in total)
  • CANN：25.3.rc1
  • Communication NIC：enp189s0f0
  • MPI：MPICH 3.2.1

8. Common Issues

8.1 Prerequisite Issues

Issue Cause Solution
SSH passwordless login failed SSH key not configured Run ssh-copy-id to configure passwordless login
Inconsistent CANN versions Different CANN versions across multiple machines Unify the CANN version across all nodes
NPU Alarm status NPU hardware failure Check npu-smi info -t health to exclude faulty cards

8.2 Runtime Issues

Issue Cause Solution
gethostbyname failed Hostname cannot be resolved Configure /etc/hosts
retcode: 7 Residual process interference Run cleanup command:mpirun -f hostfile -n 16 pkill -9 -f "all_reduce_test"
retcode: 5 Incorrect variable-length parameter configuration AllGatherV/ReduceScatterV require special parameter configuration

8.3 Other Notes

  1. Docker container: If testing with a Docker container, the host network mode must be used
  2. Log viewing: When a test fails, you can view ~/ascend/log/debug/plog the latest logs in
  3. Process cleanup: Before testing, check whether other processes are occupying the card; clean them up manually if necessary

For detailed troubleshooting, see references/common-issues.md

9. Scripts

9.1 Pre-check script

# 多机测试前置检查
./scripts/pre-test-check.sh 175.99.1.2 175.99.1.3

Check contents:

  • SSH passwordless login
  • CANN version consistency
  • NPU health status
  • Network connectivity

9.2 Quick verification script

# 测试核心算子
./scripts/quick-verify.sh 8

# 完整性能测试
./scripts/quick-verify.sh 8 full

9.3 Multi-machine test script

# 一键多机测试（自动前置检查 + 测试）
./scripts/multi-node-test.sh --nodes 175.99.1.2,175.99.1.3 --npus 8 --mode full

Official References