external-gitcode-ascend-vllm-ascend-deploy

vllm-ascend-deploy

One-click deployment tool for vLLM large model inference services on the Ascend NPU platform.

Core workflow: SSH check → parse model → NPU check → configuration discovery → user confirmation → execute deployment → start monitoring → verify service

Pre-deployment preparation

1. SSH connection check (for remote deployment)

ssh -o BatchMode=yes <server> "echo ok"

When the connection fails:

Ask the user for the SSH password
Use paramiko to configure passwordless login
Continue after verification succeeds

2. Ask the user for configuration (preferred) ⚠️

First ask the user:

Is there a specified container image path?
Is there a specified model weight path?
Are there specific startup parameters?

Judgment criteria:

User reply	Meaning	Follow-up action
Specific path/version	Explicitly provided	Use directly, skip search
"You decide", "whatever"	Vague authorization	Must search documentation to verify
No reply/skipped	Not provided	Must search documentation to verify

Prohibited: Treating vague authorization as explicitly provided and skipping the verification step

3. NPU resource check

npu-smi info          # 检查 NPU 状态和卡数
npu-smi info -t pid   # 检查占用进程

Occupancy handling: NPU occupied → list processes → ask the user (kill process/wait/cancel)

4. Configuration discovery ⚠️ Must be performed

Key rules (for the complete process, see model-discovery.md）：

Rule	Description
Deep search of image repository	Documentation examples usually show only one architecture (such as A3),must enter the image repository directory to view the complete file list
NPU architecture determination	`npu-smi info` shows 8×910B3 = A2，16×910B3 = A3
Image matching	A2 uses the `-a2` suffix, A3 uses the `-a3` suffix; if they do not match, startup may fail
Confirmation checklist	First show the officially recommended configuration, then inform the user of existing resources and let the user choose

Information source priority:

Priority	Source	URL
1️⃣	GitCode tutorial	https://gitcode.com/org/vLLM_Ascend/
2️⃣	Official documentation	https://docs.vllm.ai/projects/vllm-ascend-cn/zh-cn/latest/tutorials/models/
3️⃣	Modelers online library	https://modelers.cn/user/vLLM_Ascend

Extract content: image + TP recommended value + startup parameters + environment variables

When the network is restricted: Use the browser standalone browser tool to access; do not guess parameters

Prohibited:

Drawing a conclusion just because the first source was not found; you must continue searching according to priority
Using an existing container directly just because you saw it, ignoring official recommendations

5. User confirmation ⚠️ Blocking point

Prerequisite check:

The configuration discovery step has been completed
The image version has been confirmed from the documentation
TP recommended values confirmed from documentation

If any condition is not met → do not continue, return to Step 4 to search the documentation

The confirmation checklist must include:

Official recommended configuration(displayed first)
NPU architecture determination（8 × 910B3→A2, 16 × 910B3→A3）
Existing resource check(existing containers/images, for reference only)
Choice confirmation: let the user choose whether to use the official recommendation or reuse existing resources

Prohibited:

Using an existing container directly upon seeing it, ignoring the official recommendation
Not telling the user that there is a choice

Only start deployment after confirmation; never assume or guess parameters.

Execute deployment

Step 1: Load the image (if needed)

docker load -i <IMAGE_TAR_PATH>

Step 2: Create the container

bash scripts/create_container.sh \
  --mode local \
  --image <IMAGE> \
  --model-path <MODEL_PATH> \
  --container-name <CONTAINER_NAME>

Remote deployment: Add --mode remote --server <IP> --user <SSH_USER>

See deployment-procedure.md

Step 3: Start the service

Try the vllm CLI first:

vllm serve <MODEL_PATH> --tensor-parallel-size <TP> ...

When the CLI fails (e.g. ModuleNotFoundError), use the Python module:

python -m vllm.entrypoints.openai.api_server --model <MODEL_PATH> ...

Step 4: Monitor startup ⚠️ Must be performed

Problem: After starting the service in the background, exec will time out and continuous monitoring is not possible

Solution: Create a cron job to check periodically, with results sent via sessions_send to notify the user

流程：
1. 启动服务（后台）
2. 创建 cron job（每分钟检查）
3. 告知用户"启动中，完成后自动通知"
4. Cron job 检测到结果 → sessions_send 通知 → 删除 job

Create a cron job: ⚠️ Must include the complete context

The cron job runs in an isolated session with no parent session context,all information must be explicitly specified：

cron(action="add", job={
    "name": "vllm-status-check",
    "schedule": {"kind": "every", "everyMs": 60000},
    "payload": {"kind": "agentTurn", "message": """检查 vLLM 服务状态：

服务器: <SERVER_IP>
容器名: <CONTAINER_NAME>
端口: <PORT>
日志路径: <LOG_PATH>  # 通常是 /tmp/vllm.log 或其他指定路径
服务地址: http://<SERVER_IP>:<PORT>/health
开始时间: <START_TIME>

检查步骤：
1. SSH 到服务器，执行 docker exec <CONTAINER_NAME> tail -50 <LOG_PATH>
2. curl http://<SERVER_IP>:<PORT>/health
3. 成功 → sessions_send 通知用户 → 删除此 job
4. 失败 → 检查日志错误，超时则报错
"""},
    "sessionTarget": "isolated"
})

Prohibited: Omitting the log path, container name, or service address in the task message will cause the check to fail

The cron job notifies the user after detecting a result:

sessions_send(
    sessionKey="agent:main:main",  # 用户会话保持不变
    message="✅ 启动成功！服务地址：...",
    timeoutSeconds=0
)

See troubleshooting.md

Step 5: Verify the deployment

curl http://<SERVER_IP>:<PORT>/health
curl http://<SERVER_IP>:<PORT>/v1/models

Call after deployment

Service address:http://<SERVER_IP>:<PORT>

Python SDK：

from openai import OpenAI
client = OpenAI(base_url="http://<IP>:<PORT>/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="<model-name>",
    messages=[{"role": "user", "content": "你好"}]
)

Troubleshooting