SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsexternal-gitcode-ascend-large_scale_deploy

external-gitcode-ascend-large_scale_deploy

Deployment & operations

View repository

Updated 10-15 17:19

About this skill

Large-Scale Cluster Deployment

Feature Overview

A large-scale cluster automated installation and deployment tool for ascend-deployer. Helps users deploy various Ascend components across multiple nodes in a cluster environment.

ascend-deployer repository

Repository address

https://gitcode.com/Ascend/ascend-deployer

How to obtain

git clone https://gitcode.com/Ascend/ascend-deployer.git

Prerequisites

Before using this skill, ensure the following requirements are met:

  1. ascend-deployer repository: The workspace must contain the ascend-deployer repository
  2. Linux environment: The current environment must be a Linux system
  3. Inventory configuration: The user must have configured the large_scale_inventory.ini file
  4. Network access: Able to SSH access all target nodes in the cluster

Supported components

The following components can be installed:

Component Description
ascend-device-plugin Ascend device plugin for Kubernetes
ascend-docker-runtime Ascend Docker runtime
ascend-operator Ascend operator for Kubernetes
clusterd Ascend Clusterd service
copy_pkgs Package copy tool
deepseek_cntr DeepSeek container
deepseek_pd DeepSeek parallel distributed
docker_images Docker image management
driver NPU driver
fault-diag Fault diagnosis tool
firmware NPU firmware
kernels Kernel package
mindie_image MindIE image
mindio MindIO library
mindspore MindSpore framework
nnae Neural network acceleration engine
nnrt Neural network runtime
noded Ascend Noded service
npu NPU-related packages
npu-exporter NPU metrics exporter
python Python environment
pytorch PyTorch framework
resilience-controller Elastic controller
sys_pkg System packages
tensorflow TensorFlow framework
toolbox Ascend toolbox
toolkit Ascend toolkit
volcano Volcano scheduler

Usage

Script location:ascend-deployer/large_scale_install.sh

Single-component installation

cd ascend-deployer
bash large_scale_install.sh --install=<组件名>

Example - Install firmware:

bash large_scale_install.sh --install=firmware

Multi-component installation

bash large_scale_install.sh --install=<组件1>,<组件2>,<组件3>

Example - Install firmware and kernel:

bash large_scale_install.sh --install=firmware,kernels

Example - Install driver, firmware, and toolkit:

bash large_scale_install.sh --install=driver,firmware,toolkit

Workflow

Step 1: Verify prerequisites

  1. Check whether the ascend-deployer repository exists
  2. Verify whether it is a Linux environment
  3. Confirm that large_scale_inventory.ini is configured

Step 2: Verify inventory configuration

Ensure that large_scale_inventory.ini contains:

  • IP addresses of all target nodes
  • SSH connection information
  • Node groups and roles

Step 3: Select components

Help users select appropriate components based on the following factors:

  • Deployment purpose (training, inference, development)
  • Hardware configuration
  • Software requirements

Step 4: Execute installation

Enter the ascend-deployer directory and run the installation command with the selected components:

cd ascend-deployer
bash large_scale_install.sh --install=<选定的组件>

Step 5: Verify installation

After installation is complete, verify:

  • All components are correctly installed
  • Services are running on the target nodes
  • There are no errors in the installation logs

Common installation scenarios

The following commands must be executed in the ascend-deployer directory:

Scenario 1: Basic NPU environment setup

Set up a basic NPU environment on a new cluster:

bash large_scale_install.sh --install=driver,firmware,toolkit,toolbox

Scenario 2: Deep learning training environment

For deep learning training workloads:

bash large_scale_install.sh --install=driver,firmware,toolkit,pytorch,nnae

Scenario 3: Kubernetes deployment

For Kubernetes-based deployments:

bash large_scale_install.sh --install=ascend-device-plugin,ascend-docker-runtime,ascend-operator,volcano

Scenario 4: Full-stack installation

Install the complete Ascend software stack:

bash large_scale_install.sh --install=driver,firmware,toolkit,toolbox,nnae,nnrt,pytorch,tensorflow,mindspore

Scenario 5: Monitoring setup

For cluster monitoring:

bash large_scale_install.sh --install=npu-exporter,fault-diag

Inventory file format

large_scale_inventory.ini The file should follow this format:

[master]


[worker]
1.1.1.1-1.1.1.9 ansible_ssh_user="root" ansible_ssh_pass="test1234" step_len=3 test_expr="master-{ip}-{int(index)+1}-y"


[deploy_node]


[npu_node]


[large_scale]
SUB_GROUP_MAX_SIZE=5


[all:vars]

Inventory file field descriptions

Field Description
[master] Master node group
[worker] Worker node group, supports IP range notation (e.g., 1.1.1.1-1.1.1.9)
[deploy_node] Deployment node group
[npu_node] NPU node group
[large_scale] Large-scale deployment configuration
SUB_GROUP_MAX_SIZE Maximum number of nodes per subgroup
ansible_ssh_user SSH username
ansible_ssh_pass SSH password
step_len Step size
test_expr Test expression template

Troubleshooting

Common issues

  1. SSH connection failed

    • Verify that the SSH key is correctly configured
    • Check network connectivity to the target node
    • Ensure ansible_user has appropriate permissions

  2. Component installation failed

    • Check available disk space on the target node
    • Verify that the repository source is accessible
    • Check the installation logs for specific errors

  3. Service fails to start

    • Check the service status:systemctl status <服务名>
    • Check the service logs:journalctl -u <服务名>
    • Verify that the configuration file is correct

Notes

  1. Installation order: Some components have dependencies. It is recommended to install in the following order:

    • sys_pkg (system packages)
    • driver (NPU driver)
    • firmware (NPU firmware)
    • toolkit, toolbox (tools)
    • kernels (kernel packages)
    • nnae, nnrt (runtime)
    • frameworks (pytorch, tensorflow, mindspore)

  2. Parallel installation: Multiple components can be installed in a single command to improve efficiency

  3. Rollback: If the installation fails, see the rollback instructions in the ascend-deployer documentation

  4. Logs: Installation logs are usually stored in /var/log/ascend-deployer/

Related files

File Description
large_scale_install.sh Main installation script
large_scale_inventory.ini Cluster node inventory configuration
ansible.cfg Ansible configuration file
playbooks/ Ansible playbook directory