vllm-ascend-deploy

vllm-ascend-deploy

A one-click deployment tool for vLLM large model inference services on the Ascend NPU platform.

Core workflow: SSH check → parse model → NPU check → configuration discovery → user confirmation → execute deployment → start monitoring → verify service

Pre-deployment preparation

1. SSH connection check (for remote deployment)

ssh -o BatchMode=yes <server> "echo ok"

When connection fails:

Ask the user for the SSH password
Use paramiko to configure passwordless login
Continue after successful verification

2. Ask the user for configuration (preferred) ⚠️

First ask the user:

Is there a specified container image path?
Is there a specified model weight path?
Are there specific startup parameters?

Judgment criteria:

User reply	Meaning	Follow-up action
Specific path/version	Explicitly provided	Use directly, skip search
"You decide"、"whatever"	Vague authorization	Must search documentation to verify
No reply/skipped	Not provided	Must search documentation to verify

Prohibited: Treating vague authorization as explicitly provided and skipping the verification step

3. NPU resource check

npu-smi info          # 检查 NPU 状态和卡数
npu-smi info -t pid   # 检查占用进程

Handling occupation: NPU occupied → list processes → ask the user (kill process/wait/cancel)

4. Configuration discovery ⚠️ Must be performed

Key rules (see the full workflow in model-discovery.md）：

Rule	Description
Deep search of the image repository	Documentation examples usually show only one architecture (such as A3),you must enter the image repository directory to view the complete file list
NPU architecture determination	`npu-smi info` Shows 8×910B3 = A2，16×910B3 = A3
Image matching	A2 uses the `-a2` suffix, A3 uses the `-a3` suffix; mismatches may fail to start
Confirmation checklist	First show the officially recommended configuration, then inform the user of the existing resources, and let the user choose

Information source priority:

Priority	Source	URL
1️⃣	GitCode tutorial	https://gitcode.com/org/vLLM_Ascend/
2️⃣	Official documentation	https://docs.vllm.ai/projects/vllm-ascend-cn/zh-cn/latest/tutorials/models/
3️⃣	Modelers online library	https://modelers.cn/user/vLLM_Ascend

Extract content: Image + TP recommended value + startup parameters + environment variables

When the network is restricted: Use the browser standalone browser tool to access; do not guess parameters

Prohibited:

Concluding because the first source was not found; you must continue searching according to priority
Using an existing container directly upon seeing it, ignoring the official recommendation

5. User confirmation ⚠️ Blocking point

Precondition check:

The configuration discovery step has been completed
The image version has been confirmed from the documentation
TP recommended value confirmed from documentation

If any condition is not met → do not continue, return to Step 4 to search documentation

The confirmation checklist must include:

Official recommended configuration(display first)
NPU architecture determination（8 × 910B3→A2, 16 × 910B3→A3）
Existing resource check(existing containers/images, for reference only)
Choice confirmation: let the user choose whether to use the official recommendation or reuse existing resources

Prohibited:

Using existing containers directly upon seeing them, ignoring the official recommendation
Not informing the user that there is a choice

Start deployment only after confirmation; never assume or guess parameters.

Execute deployment

Step 1: Load the image (if needed)

docker load -i <IMAGE_TAR_PATH>

Step 2: Create the container

bash scripts/create_container.sh \
  --mode local \
  --image <IMAGE> \
  --model-path <MODEL_PATH> \
  --container-name <CONTAINER_NAME>

Remote deployment: Add --mode remote --server <IP> --user <SSH_USER>

See deployment-procedure.md

Step 3: Start the service

Try the vllm CLI first:

vllm serve <MODEL_PATH> --tensor-parallel-size <TP> ...

When the CLI fails (e.g., ModuleNotFoundError), use the Python module:

python -m vllm.entrypoints.openai.api_server --model <MODEL_PATH> ...

Step 4: Monitor startup ⚠️ Mandatory

Problem: After starting the service in the background, exec will time out and cannot continuously monitor

Solution: Create a cron job to check periodically, and the result is sent to the user via sessions_send to notify the user

流程：
1. 启动服务（后台）
2. 创建 cron job（每分钟检查）
3. 告知用户"启动中，完成后自动通知"
4. Cron job 检测到结果 → sessions_send 通知 → 删除 job

Create a cron job: ⚠️ Must include complete context

The cron job runs in an isolated session without parent session context,all information must be explicitly specified：

cron(action="add", job={
    "name": "vllm-status-check",
    "schedule": {"kind": "every", "everyMs": 60000},
    "payload": {"kind": "agentTurn", "message": """检查 vLLM 服务状态：

服务器: <SERVER_IP>
容器名: <CONTAINER_NAME>
端口: <PORT>
日志路径: <LOG_PATH>  # 通常是 /tmp/vllm.log 或其他指定路径
服务地址: http://<SERVER_IP>:<PORT>/health
开始时间: <START_TIME>

检查步骤：
1. SSH 到服务器，执行 docker exec <CONTAINER_NAME> tail -50 <LOG_PATH>
2. curl http://<SERVER_IP>:<PORT>/health
3. 成功 → sessions_send 通知用户 → 删除此 job
4. 失败 → 检查日志错误，超时则报错
"""},
    "sessionTarget": "isolated"
})

Prohibited: Omitting the log path, container name, or service address from the task message will cause the check to fail

After the cron job detects a result, notify the user:

sessions_send(
    sessionKey="agent:main:main",  # 用户会话保持不变
    message="✅ 启动成功！服务地址：...",
    timeoutSeconds=0
)

See troubleshooting.md

Step 5: Verify the deployment

curl http://<SERVER_IP>:<PORT>/health
curl http://<SERVER_IP>:<PORT>/v1/models

Call after deployment

Service address:http://<SERVER_IP>:<PORT>

Python SDK：

from openai import OpenAI
client = OpenAI(base_url="http://<IP>:<PORT>/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "你好"}]
)

Troubleshooting