SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslarge_scale_deploy

large_scale_deploy

Deployment & operations

View repository

Updated 10-15 19:58

About this skill

Large-Scale Cluster Deployment

Feature Overview

A large-scale cluster automated installation and deployment tool for ascend-deployer. Helps users deploy various Ascend components across multiple nodes in a cluster environment.

ascend-deployer Repository

Repository Address

https://gitcode.com/Ascend/ascend-deployer

How to Obtain

git clone https://gitcode.com/Ascend/ascend-deployer.git

Prerequisites

Before using this skill, please ensure the following requirements are met:

  1. ascend-deployer Repository: The ascend-deployer repository must be present in the workspace
  2. Linux Environment: The current environment must be a Linux system
  3. Inventory Configuration: The user must have configured the large_scale_inventory.ini file
  4. Network Access: Able to SSH access all target nodes in the cluster

Supported Components

The following components can be installed:

Component Description
ascend-device-plugin Ascend device plugin for Kubernetes
ascend-docker-runtime Ascend Docker runtime
ascend-operator Ascend operator for Kubernetes
clusterd Ascend Clusterd service
copy_pkgs Package copy tool
deepseek_cntr DeepSeek container
deepseek_pd DeepSeek parallel distributed
docker_images Docker image management
driver NPU driver
fault-diag Fault diagnosis tool
firmware NPU firmware
kernels Kernel package
mindie_image MindIE image
mindio MindIO library
mindspore MindSpore framework
nnae Neural network acceleration engine
nnrt Neural network runtime
noded Ascend Noded service
npu NPU-related packages
npu-exporter NPU metrics exporter
python Python environment
pytorch PyTorch framework
resilience-controller Elastic controller
sys_pkg System packages
tensorflow TensorFlow framework
toolbox Ascend toolbox
toolkit Ascend toolkit
volcano Volcano scheduler

Usage

Script location:ascend-deployer/large_scale_install.sh

Single-Component Installation

cd ascend-deployer
bash large_scale_install.sh --install=<组件名>

Example - Install firmware:

bash large_scale_install.sh --install=firmware

Multi-Component Installation

bash large_scale_install.sh --install=<组件1>,<组件2>,<组件3>

Example - Install firmware and kernel:

bash large_scale_install.sh --install=firmware,kernels

Example - Install driver, firmware, and toolkit:

bash large_scale_install.sh --install=driver,firmware,toolkit

Workflow

Step 1: Verify Prerequisites

  1. Check whether the ascend-deployer repository exists
  2. Verify whether it is a Linux environment
  3. Confirm that large_scale_inventory.ini is configured

Step 2: Verify Inventory Configuration

Ensure that large_scale_inventory.ini contains:

  • IP addresses of all target nodes
  • SSH connection information
  • Node grouping and roles

Step 3: Select Components

Help the user select appropriate components based on the following factors:

  • Deployment purpose (training, inference, development)
  • Hardware configuration
  • Software requirements

Step 4: Execute Installation

Enter the ascend-deployer directory and run the installation command with the selected components:

cd ascend-deployer
bash large_scale_install.sh --install=<选定的组件>

Step 5: Verify Installation

After installation is complete, verify:

  • All components are installed correctly
  • The service is running on the target node
  • There are no errors in the installation logs

Common Installation Scenarios

The following commands need to be executed in the ascend-deployer directory:

Scenario 1: Basic NPU Environment Setup

Set up a basic NPU environment on a new cluster:

bash large_scale_install.sh --install=driver,firmware,toolkit,toolbox

Scenario 2: Deep Learning Training Environment

For deep learning training workloads:

bash large_scale_install.sh --install=driver,firmware,toolkit,pytorch,nnae

Scenario 3: Kubernetes Deployment

For Kubernetes-based deployment:

bash large_scale_install.sh --install=ascend-device-plugin,ascend-docker-runtime,ascend-operator,volcano

Scenario 4: Full-Stack Installation

Install the complete Ascend software stack:

bash large_scale_install.sh --install=driver,firmware,toolkit,toolbox,nnae,nnrt,pytorch,tensorflow,mindspore

Scenario 5: Monitoring Setup

For cluster monitoring:

bash large_scale_install.sh --install=npu-exporter,fault-diag

Inventory File Format

large_scale_inventory.ini The file should follow this format:

[master]


[worker]
1.1.1.1-1.1.1.9 ansible_ssh_user="root" ansible_ssh_pass="test1234" step_len=3 test_expr="master-{ip}-{int(index)+1}-y"


[deploy_node]


[npu_node]


[large_scale]
SUB_GROUP_MAX_SIZE=5


[all:vars]

Inventory File Field Descriptions

Field Description
[master] Master node group
[worker] Worker node group, supports IP range notation (e.g., 1.1.1.1-1.1.1.9)
[deploy_node] Deployment node group
[npu_node] NPU node group
[large_scale] Large-scale deployment configuration
SUB_GROUP_MAX_SIZE Maximum number of nodes in a subgroup
ansible_ssh_user SSH username
ansible_ssh_pass SSH password
step_len Step size
test_expr Test expression template

Troubleshooting

Common Issues

  1. SSH connection failed

    • Verify whether the SSH key is configured correctly
    • Check network connectivity to the target node
    • Ensure that ansible_user has appropriate permissions

  2. Component installation failed

    • Check available disk space on the target node
    • Verify whether the repository source is accessible
    • Check the installation logs for specific errors

  3. Service fails to start

    • Check service status:systemctl status <服务名>
    • View service logs:journalctl -u <服务名>
    • Verify whether the configuration file is correct

Notes

  1. Installation order: Some components have dependencies. It is recommended to install in the following order:

    • sys_pkg (system packages)
    • driver (NPU driver)
    • firmware (NPU firmware)
    • toolkit, toolbox (tools)
    • kernels (kernel packages)
    • nnae, nnrt (runtime)
    • Frameworks (pytorch, tensorflow, mindspore)

  2. Parallel installation: Multiple components can be installed in a single command to improve efficiency

  3. Rollback: If installation fails, refer to the rollback instructions in the ascend-deployer documentation

  4. Logs: Installation logs are usually stored in /var/log/ascend-deployer/

Related Files

File Description
large_scale_install.sh Main installation script
large_scale_inventory.ini Cluster node inventory configuration
ansible.cfg Ansible configuration file
playbooks/ Ansible playbook directory