huawei-cloud-modelarts-vllm-ascend-model-deploy

ModelArts vLLM-Ascend Open-Source Large Model Deployment

Overview

Through the vLLM-Ascend inference framework combined with the online service deployment capability of the ModelArts inference platform, large language models from open-source communities such as HuggingFace and ModelScope (GLM, Qwen, DeepSeek, Llama, etc.) are quickly deployed to the Ascend NPU environment, providing an OpenAI-compatible RESTful API. This skill covers the full process: model confirmation and deployment form decision (4 forms: single-node/multi-node/PD disaggregated/PD colocated, unified single deployment unit) → weight/image preparation → startup and probe script generation → new-version online service deployment (API automatic / console manual) → OpenAI-compatible invocation verification, and has built-in ModelArts platform adaptation points (port 8080, OBS mounting, supernode affinity group, PD disaggregation, Mooncake KV transmission) and secure coding specifications (URL whitelist + user input validation).

Important Statement: This deployment solution is based on the models and images provided by the vLLM-Ascend open-source community, and is only applicable to scenarios such as functional verification and prototype testing. For the high availability, security compliance, observability, and O&M management and control requirements of production environments, do not directly use this solution for business production. Model capabilities and performance depend on the open-source community.

Core Process Overview

Step 1 Model Weight Preparation — Confirm the deployment model → Check the model documentation → Confirm the deployment form with the user (4 forms + number of nodes + replica trade-offs) → Confirm weights are ready → OBS Step 2 Deployment Image Preparation — Determine the version/tag according to the model documentation + deployment form → Confirm the SWR readiness status (ready: record the address / not ready: download it yourself or on behalf of the user) → Record the address Step 3 Script Preparation — Select the dedicated template according to the confirmed deployment form, fill in the model documentation parameters → Generate start_vllm.sh / health_check.sh → Artifact self-check gate (hard availability + soft performance, against Huawei Cloud best practices + vllm-ascend documentation) → OBS SDK upload Step 4 New-Version Online Deployment — Choose one of two: Method A API automatic (create_infer_service.py call CreateInferService) / Method B console manual (11-step click-through guidance) — deploy directly with the SWR image + OBS mounting, no longer creating an AI application (this module has been deprecated on the platform) Step 4.4 Verify Deployment(merged into Step 4) — Query service status/invocation endpoint → Confirm RUNNING Step 6 Invocation Verification — OpenAI-compatible API invocation /v1/chat/completions

Secure Coding Specifications (mandatory, all tool invocations must comply): This skill involves multiple tool invocations such as curl/scripts/API,any parameters passed to tools must be validated before being concatenated into commands, to prevent malicious input from being exploited (corresponding to security scan TM1 tool parameter abuse):

URL Whitelist Validation (mandatory): For any target URL of a curl/HTTP request,the hostname must be within the whitelistbefore execution is allowed — whitelist: docs.vllm.ai、quay.io、huggingface.co、modelscope.cn、*.myhuaweicloud.com(Huawei Cloud domains such as iam/obs/modelarts/swr),127.0.0.1/localhost(local liveness probe).URLs not in the whitelist are rejected from execution, switch to AskUserQuestion to have the user confirm the target address. User-provided URLs such as model documentation/image addresses must be used only after validating the host; directly splicing them into curl/commands is prohibited.

User input validation (mandatory): Before splicing user-provided inputs such as model names, OBS paths, SWR addresses, service names, and workspace-id into commands/request bodies, you must first performformat validation(regex/whitelist enumeration, e.g., service names may contain only lowercase letters/digits/hyphens, OBS paths start with obs:// , SWR addresses are in the form swr.<region>.myhuaweicloud.com/<org>/<name>:<tag>),Illegal inputs whose type and format do not match must be rejected directly, use AskUserQuestion to have the user provide them again; executing with defects is prohibited.

Unvalidated splicing prohibited: Directly splicing user input into shell commands/JSON request bodies is prohibited (e.g., splicing URLs into curl <url>, splicing model names into script parameters). For script-side parameter validation, refer to scripts/create_infer_service.py/scripts/upload_code_to_obs.py 's existing implementation (illegal parameters sys.exit are rejected). Key principles (common prerequisites for images and scripts): Both step 2 (image) and step 3 (script) mustfirst, based on the model confirmed by the user, locate that model's page in the vllm-ascend model adaptation documentation (https://docs.vllm.ai/projects/ascend/zh-cn/latest/tutorials/models/), extract from the page the image version/tag requirements, startup command examples, andsupported deployment forms(single-node / multi-node / PD disaggregation), and then pull the image and write the script accordingly. Guessing tags, guessing deployment forms, or applying generic commands based on experience while disregarding the model documentation is prohibited. Pre-generation validation for artifacts (mandatory; violation = artifact invalid): Before generating any deployment artifact (start_vllm.sh / health_check.sh / probe.sh / DEPLOY_GUIDE.md / deployment guidance text / API request body),you must re-Read/grep the corresponding section of this SKILL.md + templates/ the actual template files under it, taking the current on-disk content as authoritative. All deterministic values (health check failure count, startup delay, ports, probe script filenames, mount paths,TP_SIZE derivation rules, image tags, environment variable lists)must come from this skill file or the model adaptation documentation——Filling in based on experience/memory/common sense is prohibited(for example, the health check failure count is 300 as specified by the skill rather than the K8s default 3; the probe script is health_check.sh rather than the old probe.sh). If any value/filename/value in the artifact is inconsistent with the current skill file, the artifact is invalid and must be rewritten according to the skill's actual content.This skill file may be updated externally(the timestamps of SKILL.md/templates/references will change); before each generation you must check the current files, and must not assume the content is the same as in the previous session. Pre-deployment validation for API (mandatory): Before calling CreateInferService to create a service, you must check the following rules item by item; violation = the request will inevitably fail:

The request body is a top-level flat structure(name/type/group_configs[]/runtime_config placed directly at the top level)——Using service.deployment.roles[] as a wrapper is prohibited(the old skeleton reports 400 name/type/group_configs/runtime_config missing)

group_configs[].secret_type: "agency"(OBS mount delegated authentication)——dew/obspool/obs reports ModelArts.8162/8163; this field is easily overwritten and lost when modifying the request body (after correcting the port/other fields, it must be rechecked)

runtime_config.service_invoke.port must match the startup script PORT consistent (8080)—otherwise the probe detection port is misaligned

PD dual role unit_configs[].envs must be explicitly set differently PORT(e.g. Decode PORT: "7101")+ VLLM_ROLE

Health check parameters: failure_threshold=300/initial_delay_seconds=300/period_seconds=60(failure_threshold=30 insufficient window, vLLM V1 first startup of 10~20 minutes triggers Unhealthy)

runtime_config.service_limit three required items: request_timeout=300(seconds, the platform default 30s will cut off slow inference requests, reporting ModelArts.8909 request timeout: 30s—PD disaggregation first request requires prefill+decode collaboration, and a large model's first request includes weight loading/CUDAGraph compilation, so >30s will definitely trigger it),request_size_limit=20、rate_limit(e.g. {"num": 200, "unit": "SECONDS"});group_configs[].weight(100) required

token acquisition: the user-provided token may be truncated during copy (contains -/does not end with == → 401 decrypt token fail); recommended to obtain via IAM API + Python regex for complete extraction(re.search(r'(?i)^X-Subject-Token:\s*(\S+)', content, re.M), shell pipes are prohibited grep|sed|tr, long tokens will lose characters)

API calls in all environments (production/dev) must carry the X-Workspace-Id header(--workspace-id parameter)—the production environment must also carry it, it is not only the dev environment that needs it; without it, the services of the corresponding workspace cannot be found (list returns services from other workspaces or is empty), leading to a misjudgment of "no service with the same name" and creation failure (8006/8118).⚠️ The list API's X-Workspace-Id header does not perform filtering—list must use the URL query parameter ?workspace_id=<id> to filter by workspace (passing only the header returns services from the default workspace);create_infer_service.py 's cmd_list already has the query parameters appended; other scripts/manual calls to the list API must add ?workspace_id=<id>

When unsure about the request body structure: query an existing successfully deployed service (show returns version.instance_groups[].unit_configs[] which is a complete reference), construct by comparison, do not guess blindly

Deployment timeout is uniformly 300 minutes (mandatory): the top level of the CreateInferService request body must explicitly set deploy_timeout_minutes: 300(unit: minutes); when deploying from the console, fill in "Deployment timeout" as 300 minutes。Use of the platform default (20~60 minutes) is prohibited—multi-node/large models (e.g. GLM-5.2 MoE) first startup includes model loading + CUDAGraph compilation, and 60 minutes is still insufficient to reach RUNNING, which triggers a deployment timeout failure (status: FAILED, details deploy_timeout_minutes: 60). The field is at the top level of the request body (at the same level as name/type ); after constructing the request body, recheck that this field has not been overwritten or lost. The deployment form must be confirmed with the user (a new decision point in this skill): after confirming the model, you must first clarify from the model documentation/community whether the model usessingle-node deployment, multi-node deployment, or PD disaggregated deployment(step 1.1),organize the recommended form and rationale, and after confirming with the user proceed toimage and script preparation. Under different deployment forms, the startup script structure (single-machine TP / multi-machine Ray distributed / PD disaggregated dual role), health check strategy (PD distinguishes prefill/decode), and ModelArts deployment configuration (number of nodes/instances/network) differ significantly; only after determining the form can targeted adaptation be done. The deployment form is a planning-phase decision, not a runtime decision (mandatory): once the deployment form is confirmed by the user, it isFixed as the plan input, the generated startup/health check script isa dedicated artifact of this form—within the scriptno longer via DEPLOY_FORM environment variables for runtime branching. In step 3, according to the confirmed form, directly from templates/ select the corresponding template (four sets: single-node/multi-node/PD/PD+multi-node), fill in the model document parameters, and save locally in the artifact directory as start_vllm.sh/health_check.sh upload. During ModelArts deployment,there is no need and it should notpass again DEPLOY_FORM environment variables (PD's VLLM_ROLE, multi-node's NODE_RANK/MASTER_ADDR etc. are instance attributes internal to the form, injected by the deployment unit, and not subject to this restriction). ⚠️ Mandatory degradation and error-correction mechanism when official documentation is unreachable: If docs.vllm.ai the network is unreachable, it is strictly forbidden to use forum posts/blogs/search snippets as the basis for tags (these are second-hand information, and versions often do not match the model documentation).First check whether it is a TLS certificate issue, then proceed with degradation. The correct approach, in order of priority:

Security conventions when using curl to access third-party services: Enterprise/cloud environments often forward public network requests through security proxies (such as netentsec). The proxy performs TLS man-in-the-middle (MITM) certificate replacement on the target site, returning self-signed/unknown CA certificates, and by default verification fails with SSL certificate problem。Before execution, the target URL hostname must first be verified to be within the whitelist(docs.vllm.ai / quay.io / huggingface.co / modelscope.cn / *.myhuaweicloud.com / 127.0.0.1),URLs not in the whitelist are directly refused execution and the user is asked to confirm; only after whitelist verification passes may execution proceed, and it must carry -k(or --insecure) to skip the proxy MITM certificate. Verification logic: define the ALLOWED whitelist → extract the target host → grep -qw match against the whitelist → if no match, exit 1 refuse → if matched, allow execution. Note: -k only skips certificate verification,-k if it still returns 000/timeout after that, it is a real block;-k if a 200/302 can be obtained after that, it indicates it was only a certificate issue. Internal/local liveness checks (such as 127.0.0.1 health checks) do not use -k;

First verify whether it is a certificate issue: after passing the above whitelist verification, execute curl -sk -o /dev/null -w "%{http_code}" --max-time 15 to probe the target (e.g., docs.vllm.ai is in the whitelist), returns 200/302 → add -k continue checking the documentation, no degradation;

Add -k still unreachable (proxy truly blocking/network down): request the user to provide the official documentation link or page content (the user can usually access it, or provide a direct URL such as https://docs.vllm.ai/projects/ascend/zh-cn/latest/tutorials/models/<Model>.html );

Clearly disclose to the user: "Official documentation is unreachable, the current tag basis is XX, there is uncertainty, please help confirm"—it is not allowed to proceed based on guesses without disclosure;

After receiving official documentation confirmation,unconditionally switch to the official tag, and synchronously correct the startup script and the artifacts already uploaded to OBS (overwrite upload). In any case, the docker pull quay.io/ascend/vllm-ascend:<tag> explicitly marked on the model page is the final basis; tags with chip variants (such as -a3/-310p) are selected according to the target chip, and generic tags without a suffix correspond to A2 (910B) by default.

When to use this skill

The user wants to deploy open-source LLMs from HuggingFace / ModelScope to Huawei Cloud ModelArts (Ascend 910B NPU inference); needs to build a large model inference image, push it to SWR, and call the new inference API to deploy an online service; needs to configure OBS weight mounting and the startup script; needs to verify the deployed service using an OpenAI-compatible interface; needs to determine the deployment form (single-node/multi-node/PD disaggregation) or perform multi-machine distributed / Prefill-Decode disaggregated deployment

KooCLI command format standard

The core deployment operations of this skill (image push, OBS upload, calling CreateInferService) use Python scripts (scripts/) and curl by default, to support complete JSON request bodies and secure token passing. If the user's environment already has KooCLI (hcloud) configured and they wish to perform equivalent operations via CLI, the command format follows the Huawei Cloud CLI standard (installation reference references/cli-installation-guide.md)。

hcloud <Service> <Operation> --cli-region={region} [--key=value]

Service name: uppercase PascalCase, e.g. ModelArts
Operation name: uppercase PascalCase, e.g. CreateInferService
--cli-region required: specify the region, e.g. --cli-region=cn-north-4, must match the region where the resource pool is located

Example (create an online service, requires first hcloud configure configuring AK/SK or using --cli-auth-token):

hcloud ModelArts CreateInferService --cli-region=cn-north-4 \
    --project_id=<PROJECT_ID> --body='{"name":"...","type":"REAL_TIME",...}'

Note: KooCLI is an optional alternative;--cli-region is a required parameter of the CLI command. The request body fields are exactly the same as those of the Python script scripts/create_infer_service.py --raw-json (top-level flat structure), and token-sensitive credentials are still handled according to the "secure coding specification". If the hcloud command reports a parameter error, fall back to this skill's Python script approach (see step 4.3).

Constraints and limitations (confirm with the user first)

Model support list: The target model must be in the vllm-ascend official support list (model list), otherwise the user needs to verify adaptation themselves. Do not promise that an unverified model will definitely run successfully.
Ambiguous model names must list all candidates (mandatory): If the model name entered by the user is a series name/vague description (e.g. GLM5.3、Qwen、DeepSeek) rather than an exact model ID, you must first scrape the vllm-ascend model index page to searchallrelated model pages, compile a complete candidate list (weight version/hardware requirements/image tag/deployment form/download source) and present it to the user,and let the user confirm the selection,the skill is prohibited from deciding on the user's behalf which model to use(see step 1.0). The candidate list may be omitted only for an exact model ID or when the user explicitly authorizes "choose according to your recommendation".
Deployment form (new key decision): After confirming the model, you must first determine the deployment form according to the vllm-ascend model adaptation documentation——single-node deployment / multi-node deployment / PD disaggregated deployment,and confirm it with the user(see step 1.1). The basis for the determination is the model adaptation documentation (whether the deployment chapter provides multi-machine/PD examples, whether the example TP card count exceeds the single-node card count) and the model scale's hardware requirements.Once the deployment form is confirmed, it is fixed as a solution input; the script generates a dedicated version according to the form, and no longer performs runtime branching within the script through the DEPLOY_FORM environment variable. Do not assume all models are deployed on a single node by default, and do not force multi-machine/PD parameters when the documentation does not support them.
Network interconnection: Multi-node deployment and PD disaggregated deployment must ensure network interconnection between nodes, HCCL communication (multi-machine TP/DP) and KV Cache transmission in PD scenarios both depend on inter-node networking; for ModelArts, it is necessary to confirm that the resource pool supports multi-machine/distributed (e.g. RDMA/security group allowlisting).
Region/Resource Pool: It is necessary to confirm whether the region where the user account is located has an Ascend NPU dedicated resource pool, and whether the resource pool can provide multi-node (multi-machine) resources—some resource pools only have single-machine specifications, which directly limits the deployment form to single-node only.
Chip Model and Resource Specifications (Key): It is necessary to confirm the target resource pool'schip model(910B series / 910C (A3) / 310P, etc.) andnumber of cards—the startup parameters for the same model may differ under different chips/card counts (adaptation documents often give different examples by chip); chip variants have different VRAM (910B3 64GB / 910B4 32GB inference edition), which directly affects model selection, TP_SIZE, and "whether a single node can fit it"; 310P is an inference card and can only run small models below 7B, and large models will directly report errors. Confirmation method: first ask the user clearly about the chip model/card count, then consult the vllm-ascend adaptation document for the corresponding chip.
Version Compatibility: The CANN / torch_npu / vllm-ascend versions in the image must match the official compatibility table row by row, and be compatible with the Ascend driver version of the resource pool.
Production Disclaimer: The solution is only applicable to functional verification and prototype testing, and is not applicable to production commercial use.

Prerequisite Checklist

Confirm item by item with the user before starting; if anything is missing, prompt to complete it:

Item	Description	Confirmation Method
Huawei Cloud Account	ModelArts, OBS, and SWR services have been activated and real-name verification has been completed; provide OBS AK/SK and Endpoint(or temporary delegated credentials) for OBS SDK upload	User confirmation
OBS Bucket	Store model weights, startup scripts, and health check scripts; same region is recommended	User provides bucket name and region
SWR Image Repository	Store the vllm-ascend inference image	User provides organization name
Ascend NPU Resource Pool	ModelArts dedicated resource pool; need to confirmchip model(910B series/910C (A3)/310P) andnumber of cards, as well as whether the resource pool supportsmulti-node (multi-machine) specifications(determines the upper limit of the deployment form)	User provides resource pool name/chip model/number of cards/multi-machine capability
ModelArts Domain/Environment	The environment domain must be confirmed before calling the ModelArts API: Production environment `modelarts.<region>.myhuaweicloud.com` and development (dev) environment `modelarts-dev.<region>.myhuaweicloud.com` have differentworkspace lists and resource pool visibility(under the production domain, the workspace list only has default; only under the dev domain can all workspaces be found (such as test)); all API calls in subsequent step 4 (query workspace/query resource pool/create service) must use the confirmed domain	User confirmation (the user usually knows whether they are using dev or production)
Workspace Name	The ModelArts workspace to which the resource pool/service belongs (such as `default` / `test`);The workspace name must be confirmed first, then its workspace-id queried through the API(`GET /v1/{project_id}/workspaces`), and subsequent API calls carry the `X-Workspace-Id` header	User confirms workspace name + API queries workspace-id
Deployment Form + Number of Nodes	4 deployment forms (single single-node / multinode multi-node / pd PD-disaggregated / pd_colocate PD-colocated) + number of user-available nodes ≥ minimum node count (see step 1.1b; PD-disaggregated doubles resources); once confirmed, fixed as solution input, script generates specifically per form, no runtime branching	Model documentation verification + user confirmation
Model weight source	Model ID on HuggingFace or ModelScope; needs to be confirmed against the vllm-ascend adaptation documentationWeight format(e.g. BF16/W8A8) and download source	User provides model name + adaptation documentation verification
Network	Locally accessible Quay.io / HuggingFace / ModelScope (configure proxy or mirror site if necessary);Before curl accesses third-party services, first verify per the "Secure Coding Specification" that the target URL host is in the whitelist, then add `-k` to skip certificate verification(common with enterprise proxy MITM, see "Key Principles")	Actual execution verification
Tools	OBS Python SDK(`esdk-obs-python`, required for uploading deployment artifacts), obsutil (optional, for uploading weights), SWR login credentials, docker	Local environment

Interactive confirmation specification (mandatory, all user confirmation points must comply)

Core principle: In the flow,alldecision points requiring user confirmation, selection, or information provision must be completed via the AskUserQuestion interactive component(text input / enumerated selection / flow direction).Prohibitedto output plain text "please provide X"/"please confirm X" and then silently end the turn waiting for the user's reply. Usage requirements:

Enumerated/selection type: Candidates displayed directly as options (e.g. the model candidate list listed item by item as options), each option with a brief description.

Text input type: Each question has at least 2 options—provide common default/recommended values as options, or specify the required information format in the question text(e.g. "OBS weight path, format obs://bucket/prefix/model/<model name>"), the user supplements via Other/notes. Text-type information is limited to at most 4 questions at a time (tool limit); if exceeded, split into batches.

Merged questioning: Mutually independent confirmation items are merged into the same AskUserQuestion (at most 4 questions); the output of the previous step (e.g. candidate list table) must be immediately followed by an interactive inquiry, and "outputting the list then silently waiting" is not allowed. Quick reference of interactive confirmation points for each step(all must use AskUserQuestion): 1.0 model confirmation (for ambiguous names, list all candidates;secondary confirmation for experimental models)/ 1.1a deployment form (choose 1 of 4: single/multinode/pd/pd_colocate)/ 1.1b available node count + minimum node count prompt / 1.1c replica trade-off (single-replica multi-instance vs multi-replica single-instance, PD-disaggregated resources doubled)/ 1.2 whether weights are already downloaded + OBS weight path / 2.1 SWR image readiness status + address / 2.4 SWR organization name + docker login command / 3.0 chip model·card count / 3.4 OBS AK/SK/Endpoint + code directory / 4.0 ModelArts domain·workspace·region / 4.1 deployment method A/B / 4.3 IAM credentials or token acquisition method / 4.2 final whether API deploys on behalf / 6 call URL·token.

Step 1: Model weight preparation

1.0 Confirm the model to be deployed (check the documentation first, then confirm with the user; for ambiguous model names, all candidates must be listed, and making the decision on your own is prohibited)

Why must the model be confirmed before downloading? The model name in the user prompt may be a series name (such as "GLM", "Qwen") rather than an exact model ID; the same series may havemultiple model pages and variants(differing in parameter count/quantization version/weight format), so downloading directly based on a guess can easily fetch the wrong one. ⚠️ Mandatory confirmation process for ambiguous model names (core constraint): When the model name entered by the useris not an exact model ID(such as GLM5.3/Qwen/DeepSeek, rather than GLM-5.3-Flash-w8a8),you must first fetch the vllm-ascend documentation model index page (execute after validation against the whitelist in the "Secure Coding Standards" above (before accessing with curl, first verify that the URL hostname is in the whitelist, see Key Principle No. 0)), retrieve and list all relevant model candidates, and have the user explicitly select and confirm——The skill is prohibited from making the decision on the user's behalf about which model to use(it is prohibited to select based on subjective reasons such as "most commonly used/most suitable/smallest parameter count"). The candidate list may be omitted in only two cases: ① the user prompt already gives an exact model ID and the documentation has a corresponding page; ② the user explicitly authorizes "just pick the one you recommend".Do not proceed to any subsequent step before confirmation。

Parse the user prompt and determine the precision: Exact model ID(such as Qwen2.5-7B-Instruct) → record as the sole candidate, proceed to step 3; Ambiguous model name/series name → record as a candidate set,you must proceed to step 2 to retrieve all relevant model pages(taking only the first match is prohibited).
Retrieve all relevant model pages in the documentation (mandatory for ambiguous names): Filter the index page case-insensitively by series name/keyword (such as glm/qwen/deepseek) to findallmatching pages; for each page extract: weight version, hardware requirements (number of nodes/chips/cards), image tag, deployment form, download source (HF/ModelScope); organize into acomplete candidate list(table: candidate model | weight version | hardware requirements | image tag | deployment form),must include all matching items. When the index page is unreachable, handle it according to the degradation mechanism in "Key Principles"; do not list candidates from memory.
Check the weight description in the model adaptation documentation: Open the target model page and focus on reading: weightformat requirements(BF16 native / FP8 requires conversion to BF16 / W8A8 quantization),download address(HF/ModelScope repo ID)、recommended quantization version(W4A16/W8A8), whether different weight descriptions are given for different chip models,⚠️ Image version requirements (a high-frequency pitfall for experimental models): A page marked "documentation based on vX.Y.Z" is oftenhigher thanthe highest stable tag in the image repository (e.g., the GLM-5.3-Flash documentation is based on v0.30.0RC, while the highest stable version on quay.io is v0.23.0) —the image tag must be based on the Docker installation section of the model page, and you must check whether the transformers inside the image recognizes the model architecture(if not recognized, it reports Transformers does not recognize this architecture); if necessary, switch to a new team/internal image or switch to a stable-version model,docker run verify the transformers version (see 2.1 Image Type Identification).
Interactively confirm the final model with AskUserQuestion (mandatory):
- Ambiguous model name scenario (mandatory): Displaythe full candidate listcompletely in the conversation (as a table),then immediately use AskUserQuestion to list each candidate model as an option and let the user click to select; do not proceed to subsequent steps before confirmation. If resource specifications also need to be confirmed, they may be merged into the same AskUserQuestion (≤4 questions).
- Exact model ID → use AskUserQuestion to restate and confirm with the user, then proceed to 1.1; multiple variants → list each variant one by one as an option; model not in the supported list → ask whether to continue (verify on your own / switch models).
- ⚠️ Secondary confirmation for Experimental models (mandatory; violation = must not proceed to subsequent steps): If the model page is marked Experimental(such as GLM-5.3, GLM-5.3-Flash, etc.), after the user selects it and before proceeding to 1.1,you must use AskUserQuestion once more to disclose and obtain secondary confirmation——Disclosure content: ① It is an experimental version in the vLLM community, and performance optimization and reliability verification are still in progress; ② deployment success cannot be guaranteed(mismatched image versions / Transformers does not support the new architecture / startup parameters not verified / runtime exceptions, etc.); ③ the community generally does not recommend it for production.Secondary confirmation options: 继续部署(接受实验版本风险) / 放弃,换稳定版本模型; only after selecting "Continue deployment" may you proceed to 1.1, and record this confirmation in the plan (to be referenced later when reviewing failures).
- Prohibited: After outputting the candidate list table, silently waiting for the user's reply without adding AskUserQuestion.
Record the confirmation result: Record the final model ID (the one confirmed by the user), weight format, download source, and adaptation document link in the plan for use in subsequent startup script writing.

1.1 Determine the deployment form and node/replica plan (4 forms × single deployment unit) (key upfront decision)

Why must the deployment form be determined first? The same model can be deployed by the vllm-ascend community in different forms, and different deployment forms directly affect every subsequent step: image tag, startup script structure (single-machine TP / multi-machine DP / PD-disaggregated dual instances / PD-colocated single engine), health check strategy, ModelArts deployment configuration (number of nodes/replicas/supernode affinity group/inference unit).Determine the deployment form first so script adaptation is targeted, avoiding "writing a generic script first and then discovering during deployment that the form does not match and having to redo the work". ⚠️ Deployment forms converge to 4 types (unified convention of this skill): single(single node) / multinode(multi-node) / pd(PD disaggregated) / pd_colocate(PD colocated).Unified single deployment unit hosting: single node/multi-node/PD colocated = 1 inference unit (unit_configs 1 element);PD disaggregated = 2 inference units(unit_configs 2 elements: prefill + decode, within the same deployment unit).Multi-deployment-unit scenarios are no longer considered, and the 9 scenario numbers S1~S9 are no longer used. The number of replicas (single replica/multiple replicas) is decided in 1.1c based on the user's available number of nodes + reliability requirements, and is not enumerated independently.

Four-stage interactive confirmation (1.1a~1.1d, each stage uses AskUserQuestion):

1.1a Confirm deployment form (choose 1 of 4, mandatory): First check the model adaptation document (references/deployment-form-decision.md Section 4 clue table), organize the recommended deployment form + basis (model page link/content summary) for the user, use AskUserQuestion to list the 4 forms as options (single / multinode / pd (PD disaggregated) / pd_colocate (PD colocated)), and have the user click to confirm.Do not proceed to image/script preparation before confirmation。
- single (single-node): weights + KV Cache can fit on a single machine, and the document only gives single-machine examples → start_vllm.single.sh
- multinode (multi-node): the model is too large or the document gives multi-machine/DP+TP examples, and the example TP card count > single-node card count → start_vllm.multinode.sh
- pd (PD disaggregated): the document/community mentions PD, Prefill-Decode,--kv-transfer-config, Mooncake, dual-instance solution → start_vllm.pd.sh(two inference units, prefill + decode, within the same deployment unit; resources doubled)
- pd_colocate (PD colocated): the official single-engine DP form (ModelArts best practice glm_5_2.sh form), where the vLLM engine internally completes the PD division of labor → start_vllm.pd_colocate.sh
1.1b Confirm the number of available nodes + remind of the minimum node count (mandatory): Use AskUserQuestion to collect the user'stotal number of deployable nodes M(text input), and in the question disclose theminimum node count required by the current model form(single=1; multinode/pd_colocate=N=ceil(TP_SIZE/single-node card count);pd disaggregated=2×N, resources doubled). If M < the minimum node count, explicitly indicate "insufficient nodes, this form is not feasible" and ask the user to change the form or add resources.
1.1c Replica/instance trade-off (mandatory): Decide based on M——
1. By default, prioritize "single replica, multiple instances": unit_configs[].count = the number of nodes required by this unit N,group_configs[].count = 1;
2. If the user raisesreliability requirements → switch to "multiple replicas, single instance": group_configs[].count = floor(M / (N × 单元数))(number of PD disaggregated units=2, resources doubled);
3. After the decisionyou must restate it to the user via AskUserQuestion for confirmationwhether adjustment is needed.
1.1d Record the plan: Record as plan input DEPLOY_FORM(values single / multinode / pd / pd_colocate)+ NNODES(number of nodes within a single replica) + REPLICAS(replica count) + PD role assignment (if pd).DEPLOY_FORM It is only a plan record item and will not be injected as an environment variable into ModelArts or written into the script——the script directly generates a dedicated version according to the confirmed form (see step 3.2). PD's VLLM_ROLE, and multi-node master/slave determination are injected by the ModelArts deployment unit/rank table and are not subject to this restriction.

Complete decision basis and calculation rules: For the 4-form comparison table, ModelArts hierarchical concepts (deployment unit/inference unit/replica count/affinity group), the vllm-ascend clue judgment table, the minimum node count calculation, and the replica/instance trade-off algorithm, see references/deployment-form-decision.md——you must check against that file before confirmation.

1.2 Confirm whether the model weights are ready (no download on behalf)

Why not download directly? The model weight files are the responsibility of the user (or their environment) to obtain and place in OBS. This Skill does not download weights on behalf; it only confirms that the weights are ready and obtains their OBS path, avoiding unauthorized operations and the download overhead of tens of GB.

Use AskUserQuestion to interactively ask whether the download has been completedTarget model weight files (options: Downloaded / Not downloaded):
- Downloaded → Proceed to Step 2, use AskUserQuestion to collect the weight path on OBS
- Not downloaded → Based on the adaptation document weight description found in 1.0, provide the user with the model'sdownload address(specific repo ID link on HuggingFace / ModelScope), and explain the weight format requirements (e.g., native BF16 / W8A8 quantization). You may refer to scripts/download_model.py download command examples, butthe download action is performed by the user; after informing the placement format, use AskUserQuestion again to confirm the download progress/path.
Prompt the user to place the weight files in the specified OBS directory:
```
obs://{bucket_name}/{path}/model/
```
Where {bucket_name} is the OBS bucket name,{path} is the user-defined directory prefix (e.g., llm/weights). The weight files must be located directly in this directory or its subdirectories (the directory structure is the model directory structure, including config.json, etc.).
Use AskUserQuestion to collect the OBS path (text input): In the question text, specify the format requirements (e.g., "OBS weight path, format obs://bucket/prefix/model/<model name>"), provide guiding options (e.g., "Path already entered in Other" / "After selecting this option, please supplement the path"), and the user enters the actual OBS path where the weights are placed via Other/notes (e.g., obs://my-bucket/llm/weights/model/Qwen2.5-7B-Instruct),Confirm the path exists before continuing。Prohibitedplain text "Please provide the OBS path" waiting.

1.3 Verify OBS weight path

Record the OBS path provided by the user: in the form obs://{bucket_name}/{path}/model/<模型目录>。
Verify path availability(optional, if the user's environment can execute obsutil):
```
./obsutil ls obs://{bucket_name}/{path}/model/
```
- Confirm the directory contains config.json, weight shards (*.safetensors), and other key files
- If verification is not possible, the user verbally confirms the path is ready
Record the weight path in the plan: When Step 4 later calls the new inference API for deployment, fill the model source with this OBS path (read-only mounted to /home/ma-user/model)。
Determine the code directory location (deployment artifact upload target): Deployment artifacts (startup/health check/role recognition scripts, etc.) will be uploaded to the model directorysibling's code directory, i.e., obs://{bucket}/{path}/code(unless the user provides a different code directory). Record this directory as the plan input CODE_OBS_PATH, for reference during upload in Step 3.4 and deployment in Steps 4/5.

Why must weights be placed in OBS instead of locally/in the image? When ModelArts online service is deployed, weights are read-only mounted from OBS; after the service instance is destroyed and rebuilt, the weights remain, and multiple instances can share them; the image only contains the inference framework, and separating weights from the image is standard practice.

Step 2: Deployment image preparation

2.0 First check the documentation based on the confirmed model to determine the image version and tag (mandatory prerequisite)

Why must the model documentation be checked before pulling the image? Different models are adapted to different vllm-ascend versions (e.g., v0.23.0、v0.18.0rc1), and some models also havededicated images or chip variant tags(e.g., -a3-openeuler、-310p-openeuler, or even DeepSeekV4-flash-0731 model-specific tags). Blindly pulling latest will cause version/chip mismatch and startup failure.The image version/tag must be based on the requirements of the model page confirmed by the user。

Locate the model page: On https://docs.vllm.ai/projects/ascend/zh-cn/latest/tutorials/models/ , find the page corresponding to the model confirmed in step 1.0 (URL in the form .../tutorials/models/<ModelName>.html)。Before accessing with curl, first validate the URL hostname according to the "Secure Coding Standards" (docs.vllm.ai is in the whitelist), then you must add -k to skip certificate verification(enterprise proxy MITM, see "Key Principles" item 0), such as curl -sk <url>; for other documentation links provided by the user, likewise validate the host first before accessing.
- If after adding -k it is still unreachable (the proxy is genuinely intercepting, 000/timeout): do not fall back to search summaries/forum posts, follow the degradation mechanism in "Key Principles" —use AskUserQuestion to ask the user to provide the official documentation link or page content, and disclose the uncertainty to the user.
Extract image information from the model page: record the vllm-ascend versionthis model is compatible with (noted at the top of the page), the recommendedimage tag and chip variant (910B/910C(A3)/310P correspond to v0.23.0(A2)/v0.23.0-a3/v0.23.0-310p), and whether there is adedicated image; if the page does not directly give a tag, select one from the vllm-ascend official image repository tag list based on "version + chip model" (for version compatibility principles see Constraint 5);record the tag source(model page URL) for later verification.
Record confirmation: record the selected image address (source repository + tag) into the plan,then proceed to 2.1 Confirm SWR image readiness status。
If the user later provides official documentation/correct tag, correct it unconditionally: update the image address, rewrite the startup script, overwrite and upload the OBS artifact, and backfill the record.

For detailed lookup methods see references/vllm-ascend-model-doc-guide.md。

2.1 Confirm SWR image readiness status (deployment image entry decision point)

Why confirm before pulling and pushing? Pulling the image from Quay.io and then pushing it to SWR is a one-time operation with a large size (several GB). The user's environment may have already completed the download and upload of this image through another project/team image repository (the same SWR image can be reused, no need to repeatedly pull and push); conversely, if the local network cannot access Quay.io, then one can only reuse an existing SWR image or switch to a proxy. Confirming whether the image is already ready first avoids repeated pulling and pushing, saves bandwidth and time, and also avoids meaningless operations in the user's local environment.

Use AskUserQuestion to interactively ask: Have you already downloaded and uploaded the image confirmed in 2.0 (quay.io/ascend/vllm-ascend:<tag>) to the tenant's SWR image repository? (options: Ready, provide address / Not ready, pull and push yourself / Not ready, let the skill pull and push on your behalf)
Branch A — Ready (user has downloaded and uploaded): use AskUserQuestion to collect thecomplete SWR image address(text input, state the format in the question: swr.<region>.myhuaweicloud.com/<组织>/<镜像名>:<tag>, user fills it in via Other/notes):
```
swr.<region>.myhuaweicloud.com/<组织>/<镜像名>:<tag>
```
- Validation: the tag matches the tag confirmed from the model documentation in 2.0; the region matches the ModelArts resource pool region
- ⚠️ Image type identification (mandatory, especially necessary for experimental models): the image provided by the user may be anon-standard vllm-ascend image(built by the team based on the MindIE/ATB pipeline, such as when environment variables contain ATB_*/LCCL_*、SOC_VERSION=ascend910b1, WorkingDir is not /vllm-workspace, custom Entrypoint), itsstartup method may differ from the standard vllm serve is different, directly applying this skill's start_vllm.sh may cause startup failure. Handling: ask the user to confirm whether it is the standard official image (or a customization that only upgrades transformers)—if yes, proceed normally; otherwise **disclose "this image may not be the standard vllm-ascend image, and this skill's standard scripts may be incompatible"** and ask about the image's expected startup method, and if necessary switch to the image's built-in entrypoint. If it cannot be confirmed, ask the user to quickly verify on an ARM node docker run --rm <镜像> bash -c "uname -m && which python && python -c 'import transformers; print(transformers.__version__)'" (troubleshoot exec format error and transformers version).
- Record: record the full address into the solution input SWR_IMAGE_ADDRESS, and repeat it back to the user for confirmation. In the subsequent step 4 when deploying the new online service, use it directly as the model image source; this step no longer performs any pull/push
Branch B — Not ready: use AskUserQuestion to let the user choose the handling method:
- B-1 User downloads and uploads by themselves: the user completes the operations in their own environment according to the commands in 2.2 (pull) and 2.3 (push); after completion, use AskUserQuestion to collect the full SWR image address,validate and record it as SWR_IMAGE_ADDRESS(same as branch A)
- B-2 Download and upload on behalf by this skill: execute according to 2.4—use AskUserQuestion to collect SWR image repository nameand docker login command, show progress to the user during the pull/push process, and after completion return and record the full image address
- Prohibited: plain text "please provide the SWR image address" waiting.

2.2/2.3 Pull the Quay.io image and push to SWR (reference commands, branch B-1)

The commands in this section are provided as a reference for branch B-1 (user downloads and uploads by themselves); for automated execution of branch B-2 (download and upload on behalf by the skill), see 2.4.The tag must always be based on the version confirmed from the model documentation in step 2.0(it may be v0.23.0、v0.18.0rc1-a3-openeuler、DeepSeekV4-flash-0731 etc.), do not use by default latest。

# 1. 拉取(以模型文档确认的 tag 为准,示例为 0.23.0;模型专用镜像拉专用 tag)
docker pull quay.io/ascend/vllm-ascend:v0.23.0

# 2. 登录 SWR(使用 SWR 控制台"操作指南"的登录指令,形如)
# 密码经 stdin 注入(不进命令行 argv,不暴露于 ps/proc/history)
export SWR_PASSWORD="<登录密钥>"
docker login -u cn-north-4@<AK> --password-stdin swr.cn-north-4.myhuaweicloud.com

# 3. 打 tag: <region 域名>/<组织>/<镜像名>:<tag>
docker tag quay.io/ascend/vllm-ascend:v0.23.0 \
  swr.cn-north-4.myhuaweicloud.com/<org>/vllm-ascend:v0.23.0

# 4. 推送
docker push swr.cn-north-4.myhuaweicloud.com/<org>/vllm-ascend:v0.23.0

When the image is large and Quay.io pulls slowly, you can configure a proxy or a domestic image accelerator (see troubleshooting "image pull failure" in common issues); you can use docker image inspect <image> | grep -i vllm to verify that the vllm-ascend version inside the image matches the documentation
After completion, according to the requirements of branch B-1 in 2.1, provide the full address (swr.cn-north-4.myhuaweicloud.com/my-org/vllm-ascend:<tag>) to the skill for recording

2.4 Download and upload the image on behalf by this skill (executed in branch B-2)

Applicable prerequisites: the local environment can access Quay.io and docker is installed; the user has activated SWR and has obtained the login command in the console.

1. Use AskUserQuestion to collect user input (merge the two text items into the same ≤4 question, plain text waiting is prohibited): ① SWR image repository name(text input): the user provides the SWR organization name(such as my-org) andimage name(optional; if omitted, take the source image name vllm-ascend);② docker login command(text input): the login command copied by the user in the SWR console (Organization Management → Operation Guide), in the form docker login -u cn-north-4@<AK> --password-stdin swr.cn-north-4.myhuaweicloud.com(the password is injected via the SWR_PASSWORD environment variable + stdin,prohibited -p <密码> passing plaintext through argv)—the skill parses the SWR from this command Login domain/region andlogin credentials(when the user finds it inconvenient to provide the original command, provide region + AK + login key instead).

2. Execute pull and push (show progress throughout, mandatory): Run scripts/pull_push_image.sh, input parameters: source image quay.io/ascend/vllm-ascend:<tag>(confirmed in 2.0),--org <SWR组织名>、--name <SWR镜像名>(default vllm-ascend),--login "<docker login命令>". The script prints [1/4]~[4/4] step markers and passes through docker layer progress lines; set a sufficiently long timeout during execution (images are several GB, ≥ 30 minutes recommended),do not interrupt midway, and relay the execution output in sync. When pulling large images is slow, a proxy or domestic image acceleration can be configured, but the tag must still be based on the model documentation。

3. Return and record the result: when the script completes, print the full image address (in the form swr.cn-north-4.myhuaweicloud.com/my-org/vllm-ascend:v0.23.0), the skill records it as the solution input SWR_IMAGE_ADDRESS and repeats it back to the user for confirmation, for use in step 4 deployment;security reminder: remind the user to run docker logout swr.<region>.myhuaweicloud.com to clean up local login credentials.

Step 3: Startup script and health check script preparation

Why must the script correspond to the model, deployment form, and resource specifications? Different models (architecture, parameter count, context length, whether MoE) have different vllm startup parameters; the same model under differentdeployment forms(single-node / multi-node / PD separation, determined in step 1.1), differentchip models(such as 910B and 310P) and differentcard countshas greatly differing startup script structures (single-machine TP is one command / multi-machine requires Ray distributed and node roles / PD separation requires dual-role Prefill+Decode instances). You must consult the model'smodel adaptation documentation(https://docs.vllm.ai/projects/ascend/.../tutorials/models/<ModelName>.htmlin vllm-ascend), match along the four dimensions of "model + deployment form + chip model + card count", and must not apply a generic template. This is the core of this step: consult the adaptation documentation first, then write the script.

3.0 Confirm resource specifications (chip model + card count)

Before starting to consult the documentation, first use AskUserQuestion to interactively confirm the chip model and card count of the target resource pool (options lists common chip models 910B(A2)/910C(A3)/310P and various card counts, such as "910B(A2) single machine 8 cards", etc.), and record them in the solution.Chip model → product form → per-card memory quick reference table (910B3 64GB / 910B4 32GB / 910C(A3) subject to official specifications / 310P 24GB range) and card count configurations for different chips × model scales are detailed in references/ascend-hardware-matrix.md——values are subject to the vllm-ascend official adaptation documentation and Ascend official website specifications; when chip model information is missing, ask the user first, do not guess. Key point: 310P is aninference card, only suitable for small models below 7B; large models will directly error out.

3.1 Query the model adaptation documentation (locate the model page confirmed by the user)

vllm-ascend official model documentation (Chinese): https://docs.vllm.ai/projects/ascend/zh-cn/latest/tutorials/models/(the model page URL is in the form .../tutorials/models/<ModelName>.html;before accessing with curl, first validate the URL against the whitelist per the "secure coding specification")
Openthe model confirmed by the usercorresponding page, match along the four dimensions of "model + deployment form + chip model + card count", and extract (these are the direct basis for writing the startup script):
1. the vllm-ascend version adapted to this model(consistent with step 2.0, to ensure the image and script are matched)
2. Recommended startup command examples: Prefer examples matching the target chip model/card count (new version vllm serve / old version python -m vllm.entrypoints.openai.api_server, the two are equivalent)
3. Command variants related to deployment form: For multi-node, look for multi-machine/DP+TP examples and HCCL/Gloo communication environment variables (HCCL_IF_IP/GLOO_SOCKET_IFNAME/TP_SOCKET_IFNAME/HCCL_SOCKET_IFNAME、--data-parallel-size/--data-parallel-size-local/--data-parallel-start-rank/--data-parallel-address/--data-parallel-rpc-port, slave appends --headless); for PD disaggregation, look for Prefill-Decode startup parameters,VLLM_ROLE, KV Cache transfer (Mooncake) and load balancing requirements——refer to the model page/community documentation
4. Environment variables that must be set before startup: Common ASCEND_RT_VISIBLE_DEVICES、PYTORCH_NPU_ALLOC_CONF=expandable_segments:True、HCCL_BUFFSIZE、OMP_PROC_BIND=false、OMP_NUM_THREADS、VLLM_USE_V1、TASK_QUEUE_ENABLE、LD_PRELOAD=libjemalloc etc.,refer to the model page
5. vllm startup parameters: --tensor-parallel-size(corresponding to card count),--max-model-len、--dtype, quantization method (W8A8 etc.),--kv-cache-dtype, model-specific parameters (--trust-remote-code, MoE's --enable-expert-parallel etc.)
6. If the documentation differentiates by chip model (910B single-card/multi-card, 310P single-card, 910C (A3)), select the example consistent with the resource specifications
If the model is not in the supported list, inform the user that adaptation must be verified independently and usability is not guaranteed; for detailed consultation methods anddeployment form determination cluessee references/vllm-ascend-model-doc-guide.md and references/deployment-form-decision.md

3.2 Write the startup script (select the dedicated template according to the confirmed deployment form)

Principle: The deployment form has been confirmed with the user in step 1.1 and fixed as a solution input (DEPLOY_FORM)。The script directly generates a dedicated version according to the form, and does not perform runtime branching within the script via the DEPLOY_FORM environment variable——the deployment form is a planning-phase decision, not a runtime decision.

Select the corresponding template(templates/ four startup templates under , select according to the 4 deployment forms confirmed in step 1.1a):
- Single-node (DEPLOY_FORM=single) → templates/start_vllm.single.sh: single-machine single-card or single-machine multi-card (TP) direct startup, no cross-node logic; multiple replicas are carried by the ModelArts deployment replica count, and the template requires no modification
- Multi-node (DEPLOY_FORM=multinode) → templates/start_vllm.multinode.sh: cross-node distributed structure is already built in——each node runs the same script, and ModelArts injects GLOBAL_RANK_TABLE_FILE_PATH(rank table JSON path) and POD_IP(current node IP); the script calls get_master_ip.py in the same directory to read the rank table and obtain the master IP, and compares it with POD_IP to determine the master/slave role (slave starts a placeholder http server and appends --headless); according to POD_IP reverse-lookup the NIC name and bind HCCL_IF_IP/GLOO_SOCKET_IFNAME/TP_SOCKET_IFNAME/HCCL_SOCKET_IFNAME(required for multiple NICs);--data-parallel-size is dynamically derived from the total number of servers (number of nodes) in the rank table(the script calls get_master_ip.py --dp-size, no longer hard-coding default values); ModelArts needs to be configured withsuper-node affinity group instance count = N(number of nodes within a replica)
- PD disaggregation (DEPLOY_FORM=pd) → templates/start_vllm.pd.sh: instance role branching is already built in (Prefill / Decode two independent inference units, within the same deployment unit)——through VLLM_ROLE=prefill|decode(injected separately by the 2 inference units within the same ModelArts deployment unit) to distinguish roles, each setting its own port, NPU cards, and KV Cache transfer configuration; the weight directories of the two role instances are mounted to the same path; specific PD parameters (such as --deployment-config, Mooncake KV transfer, load balancing) are subject to the model page/community PD solution, through PD_EXTRA_ARGS append;PD-separated resources doubled (prefill/decode each require one set of nodes)
- PD colocated (DEPLOY_FORM=pd_colocate) → templates/start_vllm.pd_colocate.sh: aligned with the ModelArts official glm_5_2.sh single-engine DP form—all nodes run the same vLLM process group (data-parallel); the prefill/decode division of labor is handled internally by the vLLM engine, and externally there is only one API entry point, the master;no VLLM_ROLE, no Mooncake --kv-transfer-config; the master/slave roles are determined via get_master_ip.py reading the rank table to get the master IP and comparing with POD_IP (the slave starts a placeholder http server and appends --headless); look up by POD_IP to bind the NIC HCCL_IF_IP/GLOO_SOCKET_IFNAME , etc.;--data-parallel-size is dynamically derived from the rank table; users add extra parameters (such as --enable-expert-parallel, quantization parameters) directly to the MODEL_ARGS array according to the model documentation; get_master_ip.py must be copied to the artifact directory along with the startup/health check scripts; ModelArts requires configuration ofsuper-node affinity group instance count=N
Fill the template according to the content extracted from the model page in 3.1: set environment variables according to the model documentation, use the example startup command from the model documentation (including model-specific parameters), and check the "Usage Instructions" at the top of the template item by item.
Save as artifact: in the local artifact directory ./<模型名>-<部署方式>/ save as start_vllm.sh(the template file name is only for identifying the form; artifacts are uniformly named start_vllm.sh for ModelArts reference).
Common key points (independent of the deployment form, already built into all templates):
- Read from environment variables MODEL_NAME、TP_SIZE、MAX_MODEL_LEN , etc., to facilitate overriding via environment variables during ModelArts deployment
- TP_SIZE must be filled with a fixed value (mandatory): TP_SIZE the default value must be thefixed numberextracted from the model adaptation documentation (such as 2/4/8),deriving dynamically from ASCEND_RT_VISIBLE_DEVICES is prohibited—ModelArts-injected ASCEND_RT_VISIBLE_DEVICES may provide only a single card (such as "0"), which would be derived as TP=1; large model weights cannot fit into a single card's VRAM, KV cache becomes negative → startup crash ValueError: No available memory for the cache blocks; the template has been changed to a fixed default value, and when generating artifacts replace it according to the model documentation --tensor-parallel-size ;self-check H12 mandatory verification that TP_SIZE is a fixed number and is consistent with --tp-size (FAIL blocks upload)
- Explicitly set ASCEND_RT_VISIBLE_DEVICES
- set the necessary Ascend environment variables according to the model documentation (such as PYTORCH_NPU_ALLOC_CONF、HCCL_BUFFSIZE、OMP_PROC_BIND , etc.)
- Model-specific parameters (quantization,--enable-expert-parallel、--trust-remote-code , etc.) are appended via the template's built-in MODEL_ARGS bash array(one element per item, JSON wrapped in single quotes)—using string variables (such as EXTRA_ARGS) to store JSON parameters is prohibited: ${EXTRA_ARGS} ; when expanded the quotes are lost and argparse reports error: argument --speculative-config: Value '{"method": cannot be converted .... The EXTRA_ARGS environment variable can still be used to appendordinary parameters without JSON(automatically merged into the array within the template)
- The script must be executed with bash (critical): because the script uses bash arrays, the ModelArts startup command must use bash Execution — at deployment time (the code directory has already been file-storage mounted to /home/ma-user/code/) the startup command is bash /home/ma-user/code/start_vllm.sh;Do not use sh to execute(sh/dash does not recognize (...) arrays, reports Syntax error: "(" unexpected)
- The health check failure count is automatically set to 300 (to avoid a restart loop): The vLLM V1 engine requires compiling IR + capturing CUDAGraph on first startup, which takes 5~10 minutes; at deployment time the health checkfailure count (failureThreshold) is automatically set to 300(period 60s ≈ 18000s/300-minute tolerance window), combined with a startup delay of 600s, completely avoids probe false kills causing repeated instance restarts (see step 4.2 step 8 and the common troubleshooting "service repeatedly restarts")
- Model weight mount self-check (mandatory): Before executing the startup command, you must check whether ${MODEL_PATH}/config.json exists; if missing, print the contents of the mount directory and exit (see the code block below).Why? When the weights are not correctly mounted, vllm/transformers will treat the local path as a HuggingFace repo id, reporting the obscure error OSError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '<MODEL_PATH>', which is extremely hard to diagnose; checking in advance can directly print the root cause (wrong OBS storage address level, mount path inconsistent with MODEL_PATH, missing OBS authentication credentials causing an empty mount)
- File references within the script must be located relatively: When the script references files in the same directory (such as calling get_master_ip.pyin multi-node scenarios), always use SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) for locating,hardcoding is prohibited /code/ and other absolute paths。Why? The path where the user mounts the script may not be /code/(depending on their OBS file storage mount configuration); hardcoding will cause the probe to report can't open file '/code/xxx.py': No such file or directory
- Listen on port 8080 (or consistent with the subsequent health check), providing an OpenAI-compatible interface

The template is authoritative; before generating artifacts you must Read the actual template file: For a complete runnable startup script template, see templates/start_vllm.single.sh(single-node) / start_vllm.multinode.sh(multi-node) / start_vllm.pd.sh(PD disaggregated) / start_vllm.pd_colocate.sh(PD colocated); this skill does not copy example code within SKILL.md —before generating artifacts, re-Read/grep the template file according to "artifact generation pre-check", and fill in based on the current on-disk content(bash array MODEL_ARGS, weight mount self-check, relative path locating,DATA_PARALLEL_SIZE dynamic derivation, Mooncake semi-structured parameters, etc. are all already built into the template).

3.2.5 ModelArts platform adaptation (from the official vllm-ascend script to ModelArts adaptation)

Why is platform adaptation needed? The startup commands given in the official vllm-ascend model adaptation documentation are designed for bare metal/container direct runs (local paths, port 8000, direct CLI parameters) and cannot be used directly for ModelArts online service deployment. Adaptation checklist (check item by item after template filling is complete, see references/modelarts-console-guide.md Section 10): The startup command is packaged as a bash script (exec python -m vllm.entrypoints.openai.api_server); listening port 8080(consistent with the probe); model path ${MODEL_PATH:-/home/ma-user/model}(OBS read-only mount); parameters use a bash array MODEL_ARGS(JSON single quotes);DATA_PARALLEL_SIZE dynamically derived from the rank table(the script calls get_master_ip.py --dp-size, no hardcoding); health check: all nodes probe master /health (three-layer probes in single-engine form); probe configuration: initial delay 600s / timeout 60s / period 30s / failure threshold 300; startup command bash /home/ma-user/code/start_vllm.sh(sh prohibited); file references within the script $(dirname "$0") relative positioning (no hardcoding /code/); weight self-check config.json; multiple replicas are carried by the deployment replica count; supernode affinity group = N;PD separation injects 2 inference units within a single deployment unit VLLM_ROLE + different PORT; Mooncake via MOONCAKE_* environment variables → PD_JSON_ARGS automatically assembles --kv-transfer-config。

3.3 Write the probe script (command check method, recommended for new online services)

New online service health check recommendedexecute command checkmethod (per official site convention: startup probe bash /code/health_check.sh startup, readiness probe bash /code/health_check.sh readiness, liveness probe bash /code/health_check.sh liveness). Consistent with the startup script, the probe scriptdoes not use DEPLOY_FORM environment variables for runtime branching—it naturally distinguishes single/multi-node by "whether the artifact directory contains get_master_ip.py" (in multi-node/PD colocation scenarios the script is copied during generation; in single-node/PD separation it is not copied).

Probe behavior for each deployment form: single node (single)startup/liveness checks the vllm process,readiness probes local /health; multi-node (multinode) and PD colocation (pd_colocate, same single-engine DP form)startup/liveness checks the vllm process,readiness calls get_master_ip.py to get the master IP, and all nodes probe the master's /health(unified external entry point); PD separation (pd) has each role probe according to the injected PORT probe /health。

General points: ports are consistent with the startup script (default 8080, PD by role PORT); references to files in the same directory within the script must use $(dirname "$0") relative positioning, no hardcoding /code/(otherwise it reports can't open file '/code/xxx.py'); in multi-node scenarios get_master_ip.py must be copied and uploaded together with the script (see 3.4); three-layer probe semantics (K8s standard): startup prevents startup false kills,liveness restarts the container on failure,readiness removes from traffic endpoints on failure.The template is authoritative: for a complete runnable script see templates/health_check.sh; before generating artifacts, re-Read/grep the template file according to "pre-generation artifact validation".

3.4 Upload deployment artifacts to OBS (using the OBS SDK)

Upload the adapted generatedall deployment artifactsto the OBS code directory, including: startup script (start_vllm.sh), probe script (health_check.sh), role identification script (for multi-node scenarios, get_master_ip.py , etc.) and other files referenced within the code directory. Multiple roles/instances of multi-node and PD separation share the same code directory (each node in multi-node runs the same script; the Prefill/Decode instances of PD are distinguished by VLLM_ROLE to distinguish roles).

Local artifact directory naming convention (mandatory): the adapted artifact is underthe current working directorynamed as <模型名>-<部署方式> as a directory (e.g. Qwen3-8B-pd), no longer using the generic code。<模型名> take the user-confirmed model ID;<部署方式> identified by a lowercase suffix, corresponding to DEPLOY_FORM correspond: single node=single(Qwen3-8B-single, single-machine multi-GPU can be written as tp{N}), multi-node=mn(Qwen3-397B-mn), PD separation=pd(Qwen3-8B-pd), PD colocation=pd_co(DeepSeek-V3-pd_co). Other deployment methods can have custom descriptive lowercase suffixes; when multiple deployment methods of the same model coexist, each has its own independent directory and they do not overwrite each other.

Local aggregation artifacts (per naming convention): The start_vllm.sh、health_check.sh saved after filling in the dedicated template selected per 3.2/3.3, placed into the ./<模型名>-<部署方式> directory under the current working directory (e.g., ./Qwen3-8B-pd、./DeepSeek-V3-pd_co);Multi-node (multinode) and PD co-located (pd_colocate) deployments also require copying scripts/ from get_master_ip.py into this directory; execute chmod +x to grant execute permission.⚠️ LF line endings mandatory (required; violation = bash cannot execute): Artifacts and templates must be saved/uploaded with LF line endings—editing/copying on Windows will introduce CRLF, and bash will report $'\r': command not found / syntax error near unexpected token '$'{\r''(startup probe fails and turns FAILED). Before uploading, must verify: python -c "d=open('<file>','rb').read(); assert b'\r\n' not in d, 'CRLF found'"; if CRLF is detected, first convert to LF (d.replace(b'\r\n', b'\n')) then upload. 1.5. ⚠️ Artifact self-check gate (mandatory; FAIL means must not upload): After artifacts are generated and before uploading, must follow the references/artifact-check.md self-check checklist and run scripts/check_artifacts.py to statically verify the artifacts'availability (hard requirement)andperformance optimality (soft requirement)—based on the confirmed model/resource specifications/deployment form, against the **Huawei Cloud official best practices (modelarts_10_3121)** and the vllm-ascend model adaptation documentation:
```
python scripts/check_artifacts.py --artifact-dir ./<模型名>-<部署方式> \
    --deploy-form <single|multinode|pd|pd_colocate> \
    --doc-params '<从模型文档提取的推荐参数 JSON>' \
    --model-doc-url <模型文档 URL> \
    --tp-size <卡数> --dp-size <节点数>
```
- Availability hard checks (H01~H12): files complete/LF line endings/bash syntax/bash arrays/weights self-check/port consistency/probe command/multi-node and PD adaptation/relative paths/model documentation required parameters/TP_SIZE fixed value consistent with documentation——any FAIL prohibits upload, rerun after fixing until all pass;
- Performance soft checks (S01~S04): documentation performance parameter completeness/Ascend environment variables/TP-DP matching/deployment configuration recommendations——WARN does not block upload, but mustdisclose prompt itemsto the user (e.g., if FULL_DECODE_ONLY graph mode is not enabled, missing enable_cpu_binding, missing MTP, etc.), and the user decides whether to optimize and regenerate;
- --doc-params Source: startup command parameters extracted from the vllm-ascend model page in step 3.1 (e.g., --speculative-config/--compilation-config/--additional-config/--quantization , etc.); if not provided, the script prompts but does not FAIL;skill process requires it to be provided(the self-check comparison depends on it).
Use AskUserQuestion to interactively confirm the upload target directory (two methods as options):
- Method A — provided by user: The user directly provides the OBS code directory, e.g., obs://<bucket>/<prefix>/code(text input; state the format in the question, and the user fills it in via Other/notes);
- Method B — derived from the model directory (recommended): the user has already provided the model directory obs://<bucket>/<prefix>/model/<模型名>, createat the same levelthe code directory obs://<bucket>/<prefix>/code(consistent with the ModelArts official best practice directory structure:model/ at the same level as code/ ).Note: the OBS target directory is fixed as code, regardless of the local ./<模型名>-<部署方式> directory name(The local artifact directory name is only for local distinction; when uploading, use --code-dir specified by the parameter).
Use AskUserQuestion to collect OBS AK/SK/Endpoint (text input; state the format in the question; credentials are sensitive, remind the user they can be provided via environment variables; merge the three values into one AskUserQuestion): run scripts/upload_code_to_obs.py, the command is as follows:
```
export OBS_AK="<AK>"
export OBS_SK="<SK>"
export OBS_ENDPOINT="https://obs.<region>.myhuaweicloud.com"
# 方式 A: 用户提供 code 目录(--code-dir 为本地 ./Qwen3-8B-pd)
python scripts/upload_code_to_obs.py --code-dir ./Qwen3-8B-pd \
    --code-obs-path obs://<bucket>/<prefix>/code
# 方式 B: 由模型目录推导同级 code 目录
python scripts/upload_code_to_obs.py --code-dir ./Qwen3-8B-pd \
    --model-obs-path obs://<bucket>/<prefix>/model/Qwen3-8B
```
- The script is based on the OBS Python SDK (esdk-obs-python) recursively uploads the entire local artifact directory (all artifacts such as startup/health check/role identification scripts), prints each uploaded object and the final code directory address
- If the user's environment cannot install the OBS SDK and they insist on using obsutil, you may equivalently run ./obsutil cp ./Qwen3-8B-pd/ obs://<bucket>/<prefix>/code/ -f -r, butuse the OBS SDK by default
Record the code directory (required): record the upload target as the solution input CODE_OBS_PATH(e.g. obs://<bucket>/<prefix>/code), and repeat it back to the user for confirmation. In step 4, when deploying the new online service, the startup script path, probe command, and role identification script references are all based on this directory.

Step 4: Deploy the new online service (API automated / console manual)

Why use the new deployment? The ModelArts platform's two-step process of "create AI application + deploy online service" (old version) has been deprecated and taken offline. The new online serviceno longer creates an AI application, and directly specifies the SWR image + OBS weight mount + probe command + replica/role structure in the deployment configuration (request body structure service.deployment.roles[], corresponding to deployment replicas and inference units).

4.0 Prerequisite: Confirm solution inputs

Before deployment, confirm that the following solution inputs are complete (all produced by previous steps): SWR_IMAGE_ADDRESS(inference image address, step 2) / CODE_OBS_PATH(code directory, containing start_vllm.sh/health_check.sh/get_master_ip.py, step 3.4) / MODEL_PATH(weight OBS directory, read-only mounted to the container /home/ma-user/model, step 1.3) / DEPLOY_FORM + NNODES + REPLICAS(single/multinode/pd/pd_colocate + number of nodes per replica + number of replicas, step 1.1) / resource pool information(name, chip model, number of cards per replica, whether multi-node is supported, step 3.0) / MA_ENDPOINT(ModelArts domain, production modelarts.<region>.myhuaweicloud.com or dev modelarts-dev.<region>..., must be consistent with the resource pool environment) / WORKSPACE_NAME + WORKSPACE_ID(API calls carry the X-Workspace-Id header, check the ID in step 0 of 4.3).

⚠️ Before calling the ModelArts API, you must first confirm the environment information (mandatory): You must first interactively confirm the ModelArts domain (production / dev), workspace name, and region via AskUserQuestion before calling any ModelArts API(merge the 3 items into the same AskUserQuestion). Different domains → different workspace list/resource pool visibility → you must use the correct domain + correct X-Workspace-Id。Prohibitedcalling the API directly with the default domain (production) without confirming the environment,and also prohibitedwaiting with plain text "please confirm the environment".

4.1 Choose a deployment method (API automated / console manual)

Use AskUserQuestion to interactively confirm the deployment method(options: Method A — API automated deployment / Method B — console manual deployment, with applicable scenario descriptions); the two methods are equivalent, choose according to user preference:

Method	Applicable scenario	Features
Method A — API automated deployment	The user is willing to provide an IAM Token and wants the skill to call the API on their behalf to complete the deployment	Complete with one command, requires `--endpoint`/`--project-id`/token (environment variable `IAM_TOKEN` or `--token-file`)
Method B — Manual console deployment	The user cannot conveniently provide a Token, or wishes to retain manual confirmation for each configuration item	The skill will print the complete operation steps directly in the conversation(including the specific values for that model), and the user follows these to click through the operations in the console

Method B output requirements (mandatory): After the user selects Method B, the skill must print the complete operation steps of 4.2 one by one into the conversation, and replace all placeholders with the plan inputs from 4.0, generatingself-contained, directly actionablestep-by-step guidance.ForbiddenOnly providing a file path or "see references/xxx" for the user to consult on their own. After deployment is complete, return to 4.4 to verify service readiness and Step 6 call verification.

4.2 Method B: Manual console deployment (step-by-step click guidance)

Method B does not request API environment information (mandatory): After the user selects Method B,must not ask the user for the ModelArts domain (production/dev), workspace name, resource pool name, project-id, IAM Token, and other parameters needed only for API deployment (Method A)—during console deployment, these are all selected directly by the user on the console page. Before printing the guidance, only confirmthe inputs required by Method B itself: service name (can be auto-generated or filled in by the user), deployment form/resource specification (confirmed in steps 1.1/3.0).The only exception: After printing the complete guidance, proactively ask once, as required at the end of this subsection, "whether API-assisted deployment (Method A) is needed"—that is offering an option, not requesting parameters. This subsection is a print output template, not a document for the user to consult. After the user selects Method B, the skill follows references/modelarts-console-guide.md the 11-step structure in Section 1and prints it completely into the conversation, replacing all placeholders ({CODE_OBS_PATH}/{MODEL_PATH}/SWR_IMAGE_ADDRESS/service name/scenario values, etc.) with the actual values from the 4.0 plan inputs, outputting step-by-step guidance specific to that model. The console path is subject to the actual version (new public cloud entry: left-side menu Model Inference > Online Inference; in some versions it is Model Deployment > Online Service)。 Print format (mandatory): Start each step with **第 N 步 — <标题>** (1 Enter deployment page / 2 Basic information / 3 Resource configuration / 4 Model configuration and file storage mounting / 5 Unit configuration / 6 Deployment management configuration / 7 Fault recovery / 8 Health check / 9 Runtime environment configuration / 10 Affinity scheduling / 11 Confirm deployment), and below it list the specific actions to be performed in that step andthe specific values already filled in(not placeholders, not descriptions of the "select according to scenario" type); when printing, directly expand the differences among the 4 deployment forms (see 4.5 and the reference quick-reference table) into the definite values corresponding to that form. The entire output must be self-contained—the user does not need to open any other file. Key value quick reference: service name in lowercase letters/numbers/underscores; weight mounting /home/ma-user/model; code mounting CODE_OBS_PATH → /home/ma-user/code/; image SWR_IMAGE_ADDRESS(tag subject to the model documentation, do not use latest); container port 8080; probe command bash /home/ma-user/code/health_check.sh startup|readiness|liveness; startup delay 600s, failure count 300, timeout 60s, period 30s; startup command bash /home/ma-user/code/start_vllm.sh(sh disabled); PD unit name lowercase prefill/decode; multinode/pd_colocate affinity scheduling strongly affinitive pointing to the same super node (number of affinity group instances = N); pd (PD disaggregation) injects 2 inference units within a single deployment unit VLLM_ROLE=prefill|decode, and provided when required by the model documentation MOONCAKE_MASTER_ADDR。 Step 9 environment variables single source (mandatory): only list the items required by the "requires handling" cell in column 4 of the corresponding scenario row of the 4.5 table — if that column says "no injection required" (S1~S5), then directly write "no environment variables need to be filled in";do not expand the script's default parameters on your own(TP_SIZE/MODEL_PATH/PORT/Ascend environment variables/MODEL_NAME etc. are not listed at all; the script already has defaults or automatically derives them).The deployment form is already fixed in the script, so there is no need and it should not be passed again DEPLOY_FORM environment variables。 File storage mount must be printed in full (mandatory): when printing in step 4must include the "file storage mount" sub-item— the code directory (CODE_OBS_PATH actual value) is mounted to /home/ma-user/code/, displayed alongside the weight mount (/home/ma-user/model), and must not be omitted. Omission will cause the startup/probe scripts to be unable to access it (error can't open file)。 Output self-contained (mandatory): when printing each step, replace all placeholders with the determined values of that model;references/modelarts-console-guide.md and references/deployment-form-decision.md are only for the skill's reference when filling in values,do not push to the user。 After displaying the console guidance, proactively ask about API deployment (mandatory): after outputting the complete 11-step console guidance,you must immediately use AskUserQuestion to interactively ask: whether the skill should directly deploy on the user's behalf via API (method A)? (options: deploy on my behalf, provide IAM Token / operate in the console yourself). If the user agrees, execute according to method A in 4.3; if the user chooses to operate in the console, then end the delivery.Do not default to the user operating the console themselves, and do not default to deploying on the user's behalf — you must ask;Prohibitedto output the guidance and then wait silently without asking.

4.3 Method A: call create_infer_service.py to deploy (API automatic)

Method A prerequisites: requires --endpoint(ModelArts service endpoint, in the same region as the resource pool),--project-id, token (request header Authorization: Bearer <token>, valid for 24h),--workspace-id(optional, the dev environment requires the X-Workspace-Id header).Environment information (domain/region/workspace) has already been confirmed in the "interactive confirmation specification" or in 4.0 using AskUserQuestion; IAM credentials are collected using AskUserQuestion (see token acquisition and extraction below)。project-id no need to separately request from the user— after obtaining the token, use GET https://iam.<region>.myhuaweicloud.com/v3/auth/projects(with the X-Auth-Token header) to automatically query, match the region by name (e.g. cn-north-7) and take id (see step ⑥ of step 0). ⚠️ Secure token transmission (mandatory, secure coding specification): the IAM Token is a sensitive credential,prohibited from being passed as a command-line argument(ps//proc/<pid>/cmdline/Shell history will expose it in plaintext). The script has removed the --token parameter; provide it in order of priority: ① environment variable IAM_TOKEN(recommended, does not enter the process argv; on Windows always use this method — NTFS permission mapping 0o666,chmod 600 is unreliable,--token-file will be rejected) ② --token-file <文件>(reliable only on Linux; permissions must be 600; the script rejects 0o644, etc.) ③ Interactive input(no echo, cannot be piped/scripted). ⚠️ Agent Bash tool has an independent shell: each invocation of the Agent Bash tool is an independent shell,export IAM_TOKEN=... does not persist across invocations——must IAM_TOKEN=<token> python scripts/create_infer_service.py ... be completed within the same command (the token can be python -c "...读 token 文件..." read inline and assigned to an environment variable before execution).⚠️ Extremely long token (base64-encoded PKI tokens can reach 8KB+): passing via bash heredoc/inline will fail with errors due to quote/special character conflicts unexpected EOF——Do not use bash heredoc/cat <<EOF to pass the token; instead, use the Write tool to write the complete Python script containing the token into a .py file (with the token as a Python string constant), then use python <脚本>.py to execute it (within the script, requests directly references the token variable, without going through shell argv). ⚠️ Step 0: Before calling any ModelArts API, environment information must first be confirmed (mandatory): do not call directly with the default domain (production modelarts.<region>.myhuaweicloud.com). ① Use AskUserQuestion to interactively confirm the domain (production modelarts.<region>.myhuaweicloud.com / development modelarts-dev.<region>.myhuaweicloud.com——under the production domain the workspace list returns only default; only the dev domain can find all workspaces); ② Use AskUserQuestion to confirm the workspace name (options + Other); ③ Use the confirmed domain to query workspace-id (GET /v1/{project_id}/workspaces, with Authorization: Bearer <token>,no X-Workspace-Id header is needed), match by workspace name to obtain the id recorded as WORKSPACE_ID(if not found → switch to the dev domain and query again); ④ to query the resource pool list use GET /v2/{project_id}/pools(the v1 path /v1/{project_id}/pools returns 404 APIGW.0101)——from the response items[] take metadata.name as the resource pool ID (pool_id),spec.resources[].count is the number of nodes,must include the X-Workspace-Id header; to query the number of cards on a node use GET /v2/{project_id}/pools/{pool_id}/nodes(response items[].status.resources["huawei.com/ascend-1980"]=total number of cards,availableResources=number of available cards); ⑤ all subsequent API calls use: the confirmed domain + header X-Workspace-Id: {WORKSPACE_ID} + Authorization: Bearer <token>。⑥ project-id is queried automatically (no need to ask the user): after obtaining the token, call GET https://iam.<region>.myhuaweicloud.com/v3/auth/projects(with the X-Auth-Token header), in the response projects[] match by name the target region (e.g. cn-north-7) and take id which is the project_id。 ⚠️ Request body structure: the new version CreateInferService request body is atop-level flat structure——not {"service": {"deployment": {"roles": []}}} a wrapper structure (the old skeleton reports 400 name/type/group_configs/runtime_config missing).For the complete request body JSON example and field descriptions see references/modelarts-console-guide.md Section 3(top-level flat,secret_type: "agency"、service_invoke.port=8080、deploy_timeout_minutes=300(deployment timeout is uniformly 300 minutes, see API deployment pre-check item 10), PD dual roles have different PORT、unit_configs[].name only lowercase letters/digits/hyphens (uppercase Prefill/Decode reports 400),flavor is the inference instance specification (e.g. modelarts.bm.arm.192u.1536g.npu.8d910b,not the resource pool node specification modelarts.bm.npu.arm.ssd——the node specification is the ECS physical machine flavor, reports 400 ModelArts.6404 Flavor not found;The reasoning flavor format is modelarts.bm.arm.<cpu>u.<mem>g.npu.<n>d910b, derived from node status.resources 's CPU/memory/GPU-count);failure_threshold=300/initial_delay_seconds=300/period_seconds=60、group_configs[].weight(100) and service_limit are the three required fields (request_timeout=300/request_size_limit=20/rate_limit, see item 6 of the API deployment pre-check;request_timeout when missing, the platform defaults to 30s, slow/PD first request reports ModelArts.8909 request timeout)、S4 multi-node uses unit_configs[].count=N to carry the cross-node count (the number of single-replica resource instances is the max,group_configs[].count=1), the most reliable way to get the correct structure = query an existing service that was deployed successfully show and reverse-engineer it).Before constructing the request body, check against the reference; field names are subject to API Explorer。

Basic command (recommended to pass --raw-json in its entirety, zero field errors):

# 创建在线服务(单部署单元;PD 分离传两个 --role: prefill + decode,其余形式一个 --role)
# token 安全提供: 环境变量 IAM_TOKEN(推荐,不进进程 argv),或 --token-file <文件>(权限 600)
IAM_TOKEN=<IAM_TOKEN> python scripts/create_infer_service.py create \
    --endpoint https://modelarts.cn-north-4.myhuaweicloud.com \
    --project-id <PROJECT_ID> \
    --service-name qwen3-8b \
    --replicas 1 \
    --raw-json '{"name": "...", "type": "REAL_TIME", ...}'  # 完整平铺请求体见 reference

⚠️ Service name taken: creating the service returns 400 ModelArts.8006 "The service name xxx already exists" → the service name is already taken (unique within the region; it may be in another workspace or created by a previous operation of this skill).Handling order (mandatory): ① First confirm whether the occupant is "an old service holding resources" or "same name only, no resources": check the resource pool's available GPU count (see 8118 handling); ② If the resource pool's available GPU count is 0 → first stop the old service to release resources(see 8118 handling below), wait for STOPPED before creating;do not directly switch to a new service name and retry— after switching names, if the resources are still occupied by the old service, it will still report 8118 Insufficient NPU resources(Requested 16 > Available 0), wasting one more failure; ③ Only if the resource pool has sufficient GPU count and you truly need to reuse the old service name, consider switching to a new service name (lowercase letters/digits/hyphens, e.g. the original name plus -2) to recreate, or use AskUserQuestion to interactively confirm whether to manage/reuse the old service.Key: list not finding a service with the same name ≠ the service does not exist— list (GetServices) may not return non-running services such as FAILED/DELETING/STOPPING (and in the dev environment you must include --workspace-id to find the corresponding workspace's services), but that service still occupies the resource pool's GPU count; to determine whether a service exists, rely onresource pool available GPU countand the 8006/8118 returned by create .\n ⚠️ Failed/old service resource cleanup (mandatory, must check before retrying a failed deployment): Creating a new service returns 400 ModelArts.8118 Insufficient NPU resources(Requested GPU count > Available 0 GPUs), which meansthe failed old service still occupies all the GPUs in the resource pool. Handling: ① Check the old service's status GET /v2/{project_id}/services/{sid}(DEPLOYING/FAILED/STOPPING may all occupy resources); ② stop the old service to release resources POST /v2/{project_id}/services/{sid}/stop(when FAILED returns ModelArts.6850 ... can not STOP the instance has already terminated, so you can directly create a new service); ③ wait for the old service to STOPPED before creating; ④ in the dev environment DELETE may return 404 (method DELETE not found), so use stop instead.Prefer using stop rather than repeatedly trying different DELETE paths。 code directory file storage mount (key): unit_configs[].files[] configure the code directory mount within (files[].address=CODE_OBS_PATH → files[].mount_path=/home/ma-user/code); the startup command and probe commands must all use the in-container path bash /home/ma-user/code/...。 envs only needs to inject a few items: TP_SIZE no need to write into envs(the script automatically derives it based on the resource pool instance specification);MODEL_PATH/ template Ascend environment variables etc. already have default values;pd (PD separation) must be within the same deployment unit, with 2 inference units each added VLLM_ROLE=prefill|decode and PORT(dual roles differ);MODEL_NAME optional (script default value model, only injected when served-model-name needs to be customized); add when pd and the model documentation requires Mooncake MOONCAKE_MASTER_ADDR。 ⚠️ Token acquisition and extraction (mandatory): ① To obtain a token, you must use Python requests to call the IAM API: resp = requests.post('https://iam.<region>.myhuaweicloud.com/v3/auth/tokens', json=<IAM请求体>, verify=False, timeout=30)(requests passes dict/JSON directly; if credentials are correct, it returns 201), response header resp.headers['X-Subject-Token'] to get the token;It is forbidden to use curl -d to obtain a token by concatenating JSON body strings(under Windows bash, '"$VAR"' quote concatenation causes incorrect password transmission; even with correct credentials it returns 401); ② the token must be obtained with project scope("scope": {"project": {"name": "<region>"}}, in the same region as the resource pool);domain scope is forbidden(reports 400 ModelArts.6400 ... getProject() is null);③ The order in which the three IAM credential elements are requested must be "account name → IAM user name → password" (mandatory): when collecting with AskUserQuestion,display and collect in the order "account name (domain.name), IAM user name (user.name), password (password)"——the account name (primary account) is the first element users can most easily identify, so put it first; the user name (IAM sub-user) comes second; the password last. Field correspondence: account name = primary account (domain.name), user name = IAM sub-user (user.name), password = login password; wrong order will cause 401 and there is a risk of account lockout——first ask "in which way to provide the token" (IAM user name/password / directly provide Token), and after collection, restate the field correspondence to the user in the order "account name (domain.name), IAM user name (user.name), password (password)" before obtaining the token; ④ Token extraction must use Python regular expressions for complete extraction: re.search(r'(?i)^X-Subject-Token:\s*(\S+)', content, re.M)——It is forbidden to use grep | sed | tr pipes(long tokens lose characters and cause 401); ⑤ verify the projects visible to the token (optional): GET https://iam.<region>.myhuaweicloud.com/v3/auth/projects(with X-Auth-Token); ⑥ when the user directly provides a token, if it contains -/_ or does not end with == , first suspect copy truncation, and use AskUserQuestion to ask the user to choose between providing it again / switching to IAM user name and password.⑦ Ultra-long PKI Tokens are forbidden to be passed through AskUserQuestion (mandatory): IAM Tokens are in PKI format (containing a base64-encoded certificate chain + signature, up to 8KB+),the answer/notes fields of AskUserQuestion have length limits, and an ultra-long token will be truncated; the missing signature block at the end causes the API to report 401 decrypt token fail——repeated retries are ineffective, and the user may truncate it again when copying. Correct methods: ① first ask the user to provide the IAM user name/password, and the skill uses Python requests to call the IAM API to obtain the token; ② or ask the user to save the token as a local file and provide the file path, and the skill reads it and then sets os.environ['IAM_TOKEN'](environment variable method, does not enter process argv, recommended on Windows) to call the script; ③ It is forbidden to embed an ultra-long token as a Python string constant in the script(prone to quote/escape errors, and deviates from the skill's token secure transmission specification).⑧ Troubleshooting mismatched three IAM credential elements (mandatory): when credentials are correct but 401 occurs The username or password is wrong ,it is forbidden to repeatedly guess user name/password combinations(multiple errors carry a risk of account lockout)——ask the user to provide a complete postman/working request body, and check field by field user.name/domain.name/password, especially domain.name (account name) is prone to missing characters due to manual transcription/parsingand thus being mismatched (e.g. ei_pangu_x60079363_01 mistakenly written as ei_pangu_x6007963_01)。

4.4 Verify service readiness and obtain the invocation endpoint (common to both methods)

Method A (API automatic): IAM_TOKEN=<IAM_TOKEN> python scripts/create_infer_service.py list --endpoint ... --project-id ... --workspace-id <WORKSPACE_ID>(obtain service_id,All environments (production/dev) must carry workspace-id; without it, services in other workspaces or an empty result will be queried, causing a misjudgment that no service with the same name exists);... show --endpoint ... --project-id ... --workspace-id <WORKSPACE_ID> --service-id <SERVICE_ID>(obtain deployment status and invocation endpoint).⚠️ The list API response field is data[](not services[]/items[]): create_infer_service.py already follows the data field for parsing; if you use another script to call the list API, you must take resp.json()["data"], taking the wrong field will show an empty list and misjudge that no service exists.⚠️ Not found by list ≠ service does not exist: GetServices(list) may not return non-running services such as FAILED/DELETING/STOPPING, but such a service still occupies the resource pool's card count (see 4.3 "Cleanup of failed/old service resources"); to determine whether a service with the same name/an old service exists, rely on theresource pool available card countand the 8006/8118 returned by create, and do not assume it is safe to create just because list is empty.

Method B (manual via console): view the status on the online service list page, click the service name to enter the details page and obtain the "public invocation URL"(access_address)。

⚠️ dev environment invocation endpoint (predict_url): dev environment service RUNNING after show details return access_address: null, the invocation endpoint is in the predict_url[] field (e.g. "predict_url": [{"type": "PUBLIC", "urls": ["100.93.1.22"]}, ...])。Invocation URL format: http://{PUBLIC_IP}/v2/infer/{service_id}/v1/chat/completions(OpenAI compatible). Note: ① The protocol must be http(using https reports 400 ModelArts.8910 Request protocol: HTTPS does not match...); ② The path must carry the /v2/infer/{service_id} prefix (bare IP + /v1/chat/completions reports 404 ModelArts.2836 Path not found)。

General points: wait for the status to become RUNNING(deploying DEPLOYING, failed FAILED); if access_address is None, switch to predict_url[].urls[] in type=PUBLIC 's address and concatenate it in the above format; if it restarts repeatedly, first check the probe startup delay and failure count configuration (see troubleshooting "service repeatedly restarts"); the code directory (CODE_OBS_PATH) is mounted viafile storage mountinto the container at /home/ma-user/code/, startup/probe commands must all use in-container paths (bash /home/ma-user/code/start_vllm.sh、bash /home/ma-user/code/health_check.sh startup), and within scripts reference files in the same directory using $(dirname "$0") for relative location.

4.5 Mapping of 4 deployment forms → deployment configuration (API group_configs/unit_configs shared with console units)

The table below applies to both Method A (API group_configs[]/unit_configs[] fields) and Method B (console unit configuration/environment variables);For the complete mapping table and environment variable classification quick reference (① script auto-derived TP_SIZE/DATA_PARALLEL_SIZE ② in-container fixed MODEL_PATH/PORT ③ platform-injected VLLM_ROLE(pd separation only) ④ model document required MAX_MODEL_LEN/DTYPE/MODEL_NAME ⑤ community solution MOONCAKE_MASTER_ADDR) and the console quick reference table, see references/modelarts-console-guide.md Section 1 (4-form console configuration quick reference) and Section 6 (4 forms → request body mapping). Key points at a glance:

Deployment form	Inference unit	Number of replicas	Items to handle in the deployment unit/model configuration	Probe
single single-node	1 (one) `role-1`	1 (default) / M (reliability)	No injection required (`TP_SIZE` the script derives it from the card count;`MODEL_NAME` optional, the script default is `model`)	health_check.sh probes local /health
multinode multi-node	1 (one) `role-1`	1 (default) / M	Request body `unit_configs[].count`= number of nodes within the replica N(the number of resource instances in a single replica is greater,`group_configs[].count`=1); ModelArts injects `GLOBAL_RANK_TABLE_FILE_PATH`(rank table JSON path) and `POD_IP`(current node IP)—the script reads the rank table via `get_master_ip.py` to get the master IP and `--dp-size` derives `DATA_PARALLEL_SIZE`, and compares with `POD_IP` to determine the master/slave role (slave appends `--headless`)+ SuperNode affinity group=N	All nodes probe master /health
pd PD disaggregation	2: prefill + decode(within the same deployment unit)	1 (default) / M (reliability, resources doubled)	Within the same deployment unit, the 2 inference units each inject `VLLM_ROLE=prefill` and `VLLM_ROLE=decode` + different dual-role `PORT`(e.g. 8080/8081) + each `unit_configs[].count`= number of nodes for that role	Each role probes /health according to the injected PORT
pd_colocate PD colocated	1 (one) `role-1`	1 (default) / M	No need to inject any role/port environment variables—aligned with the official glm_5_2.sh single-engine DP form; the PD division of labor is handled internally by the vLLM engine; only `unit_configs[].count`=N + SuperNode affinity group=N	All nodes probe master /health

Conclusion: Requiresuser/deployment configurationThe only thing to handle is — pd (PD disaggregation) within a single deployment unit injects into the 2 inference units VLLM_ROLE and different dual-role PORT; when the model documentation requires values different from the defaults, override MAX_MODEL_LEN/DTYPE/TENSOR_PARALLEL_SIZE。multinode and pd_colocate are aligned with the official single-engine DP form, no need to inject any role variables(master/slave is determined automatically by get_master_ip.py ;DATA_PARALLEL_SIZE is dynamically derived by get_master_ip.py --dp-size );TP_SIZE has already been automatically derived by the script according to the resource specification; the number of health check failures has already been automatically set to 300 (see step 4.2, step 8); everything else has defaults or is fixed within the container, requiring no user input**. For detailed field mapping and PD multi-role configuration, see references/modelarts-console-guide.md(new API field details) and references/mooncake-pd-deploy-guide.md。

4.6 Supporting lifecycle management (Method A API automatic)

Operation	Command	Description
Scaling	`IAM_TOKEN=<IAM_TOKEN> python scripts/create_infer_service.py scale --service-id <ID> --deployment-id <DID> --replicas 2`	Adjust the number of deployment replicas (use an environment variable or `--token-file`for the token; plaintext on the command line is prohibited)
Delete service	`IAM_TOKEN=<IAM_TOKEN> python scripts/create_infer_service.py delete --service-id <ID>`	Cleanup (after testing / before rebuilding after failure)
Update configuration	New API (UpdateService/redeploy)	Refer to the official API as needed

Step 6: Call the service for verification

Once the service is ready, verify via the OpenAI-compatible API (script: scripts/test_inference.py)。The invocation endpoint is automatically obtained from 4.4 (the dev environment is http://{PUBLIC_IP}/v2/infer/{service_id}/v1/chat/completions); if the service is configured for inference request authentication, use AskUserQuestion to interactively collect the inference token (text input; state the format in the question):

python scripts/test_inference.py --url https://<modelarts-inference-endpoint>/v1/chat/completions \
    --token <token> --model <MODEL_NAME> --prompt "你好,请介绍一下你自己"
# 可选参数: --max-tokens 256 --temperature 0.7

Verification points: returns choices[0].message.content has normal text → deployment succeeded; for local integration testing, you can use http://127.0.0.1:8080/v1/chat/completions first verify the script and parameters, then go to the cloud.For the complete invocation logic, see references/verification-method.md(OpenAI-compatible invocation example, authentication handling, verification failure troubleshooting).

Output format

Outputs for each step: ① Interactive confirmation — AskUserQuestion options (model candidates/deployment form/deployment method, etc.); ② Deployment artifacts — ./<模型名>-<部署方式>/ under start_vllm.sh/health_check.sh(multi-node includes get_master_ip.py), upload to CODE_OBS_PATH; ③ Console guidance — print 11 steps step by step (replace placeholders with the determined values for that model); ④ API deployment — service id/status and invocation endpoint URL; ⑤ Invocation verification — choices[0].message.content returns text.

Best practices

model/deployment form/deployment method, etc.Confirm every decision point with the user first via AskUserQuestion, do not make decisions on the user's behalf
Image tag and startup parametersalways follow the vllm-ascend model documentation; when the official documentation is unreachable, disclose according to the fallback mechanism
Before generating artifactsre-Read/grep the template and SKILL.md(based on the current on-disk content); deterministic values (port 8080/failure count 300/startup delay 600s) must not be filled in from memory
token/AK-SK pass via environment variables or a 600-permission token-file; extract the token with Python regex, shell pipelines are prohibited
Before retrying after a deployment failure, clean up the old service resources first (when 8118 has insufficient cards, stop the old service); this solution is only suitable for functional verification/prototype testing

Common troubleshooting

For the complete troubleshooting table, see references/troubleshooting.md(locate root causes and troubleshooting actions by symptom). Quick overview of high-frequency issues:

Symptom Root cause Quick fix

Service repeatedly restarts (most common) Health check startup delay is too short; vLLM V1 first startup takes 5-10 minutes and is killed and restarted Startup delay 600s + failure count 300 (deployment config already automatically sets 300); probe path/port consistent with the script

ValueError: No available memory for the cache blocks TP_SIZE dynamically derived from ASCEND_RT_VISIBLE_DEVICES ; ModelArts injects a single card → TP=1, large model weights cannot fit into a single card's VRAM, KV cache is negative Script TP_SIZE change to the fixed value recommended by the model documentation (e.g., 2/4/8); self-check H12 enforces validation

Exits immediately on startup exec format error Image architecture is not arm64 / not a standard image (MindIE/ATB) docker inspect check Architecture; for image type identification see 2.1

Transformers does not recognize this architecture The transformers inside the image are too old and do not recognize the new architecture of the experimental model Check the model documentation version vs the image tag; switch to a newer image or a stable model

Syntax error: "(" unexpected Script is executed by sh executed Startup command uses bash /home/ma-user/code/start_vllm.sh

Repo id must be in the form... Model weights are not mounted (no config.json under MODEL_PATH) Check the OBS storage address hierarchy/mount path/authentication credentials; the script has a built-in config.json self-check

400 ModelArts.8118 Insufficient NPU resources The failed old service still occupies all cards in the resource pool ① First check the number of available cards on the resource pool nodes (GET /v2/{project_id}/pools/{pool_id}/nodes of availableResources);② POST .../services/{sid}/stop Stop the old service, wait for STOPPED before creating;If the old service cannot be found via list (non-running states are not returned), still handle it as 8118

400 ModelArts.8006 service name exists The service name is already taken (unique within the same region, including old services previously created by this skill) First stop the old service to release resources before creating (see 4.3),Do not directly change the name and retry(still 8118 after changing the name)

400 ModelArts.8006 + 8118 同现 The old service occupies the same name and resources, and list cannot find it (non-running state) The number of available cards in the resource pool is 0 → stopping the old service takes priority over changing the name; determine whether the service exists based on resource pool card count + 8006/8118

400 name/type/group_configs/runtime_config 缺失 The request body used the old skeleton service.deployment.roles[] Use the 4.3 top-level flat structure;show infer the correct structure from it

401/403 decrypt token fail IAM Token expired (24h) or truncated during copy Obtain the token again; fully extract it using the IAM API + Python regex (shell pipes prohibited)

400 ModelArts.6404 Flavor not found unit_configs[].flavor incorrectly filled in the resource pool node specification (e.g., modelarts.bm.npu.arm.ssd) fill in the inference instance specification instead (e.g., modelarts.bm.arm.192u.1536g.npu.8d910b); from node status.resources derive from the CPU/memory/card count

The workspace/resource pool list query is empty Wrong domain used (production vs dev)/insufficient permissions Confirm the environment domain;modelarts-dev.<region> only then can all workspaces be found

list returns services from other workspaces The list API's X-Workspace-Id header does not perform filtering; URL query parameters are required The list URL is concatenated with the ?workspace_id=<id> query parameter (create_infer_service.py already fixed)

Symptom	Root cause	Quick fix
Service repeatedly restarts (most common)	Health check startup delay is too short; vLLM V1 first startup takes 5-10 minutes and is killed and restarted	Startup delay 600s + failure count 300 (deployment config already automatically sets 300); probe path/port consistent with the script
`ValueError: No available memory for the cache blocks`	`TP_SIZE` dynamically derived from `ASCEND_RT_VISIBLE_DEVICES` ; ModelArts injects a single card → TP=1, large model weights cannot fit into a single card's VRAM, KV cache is negative	Script `TP_SIZE` change to the fixed value recommended by the model documentation (e.g., 2/4/8); self-check H12 enforces validation
Exits immediately on startup `exec format error`	Image architecture is not arm64 / not a standard image (MindIE/ATB)	`docker inspect` check Architecture; for image type identification see 2.1
`Transformers does not recognize this architecture`	The transformers inside the image are too old and do not recognize the new architecture of the experimental model	Check the model documentation version vs the image tag; switch to a newer image or a stable model
`Syntax error: "(" unexpected`	Script is executed by `sh` executed	Startup command uses `bash /home/ma-user/code/start_vllm.sh`
`Repo id must be in the form...`	Model weights are not mounted (no config.json under MODEL_PATH)	Check the OBS storage address hierarchy/mount path/authentication credentials; the script has a built-in config.json self-check
`400 ModelArts.8118 Insufficient NPU resources`	The failed old service still occupies all cards in the resource pool	① First check the number of available cards on the resource pool nodes (`GET /v2/{project_id}/pools/{pool_id}/nodes` of `availableResources`);② `POST .../services/{sid}/stop` Stop the old service, wait for STOPPED before creating;If the old service cannot be found via list (non-running states are not returned), still handle it as 8118
`400 ModelArts.8006 service name exists`	The service name is already taken (unique within the same region, including old services previously created by this skill)	First stop the old service to release resources before creating (see 4.3),Do not directly change the name and retry(still 8118 after changing the name)
`400 ModelArts.8006 + 8118 同现`	The old service occupies the same name and resources, and list cannot find it (non-running state)	The number of available cards in the resource pool is 0 → stopping the old service takes priority over changing the name; determine whether the service exists based on resource pool card count + 8006/8118
`400 name/type/group_configs/runtime_config 缺失`	The request body used the old skeleton `service.deployment.roles[]`	Use the 4.3 top-level flat structure;`show` infer the correct structure from it
401/403 `decrypt token fail`	IAM Token expired (24h) or truncated during copy	Obtain the token again; fully extract it using the IAM API + Python regex (shell pipes prohibited)
`400 ModelArts.6404 Flavor not found`	`unit_configs[].flavor` incorrectly filled in the resource pool node specification (e.g., `modelarts.bm.npu.arm.ssd`)	fill in the inference instance specification instead (e.g., `modelarts.bm.arm.192u.1536g.npu.8d910b`); from node `status.resources` derive from the CPU/memory/card count
The workspace/resource pool list query is empty	Wrong domain used (production vs dev)/insufficient permissions	Confirm the environment domain;`modelarts-dev.<region>` only then can all workspaces be found
list returns services from other workspaces	The list API's `X-Workspace-Id` header does not perform filtering; URL query parameters are required	The list URL is concatenated with the `?workspace_id=<id>` query parameter (`create_infer_service.py` already fixed)

Reference files

references/deployment-form-decision.md — Deployment form decision matrix (all 4 forms covered): comparison of the 4 forms, key ModelArts concepts (deployment unit/inference unit/replica count/supernode affinity group), four-stage decision process (1.1a~1.1d), model scale × hardware quick reference, replica/instance trade-off algorithm, form × step impact matrix
references/modelarts-console-guide.md — Detailed explanation of new-version online service deployment: 11-step console operation guide + 4-form console quick reference table; CreateInferService top-level flat request body JSON example + field descriptions, 4 forms → group_configs/unit_configs mapping matrix, probe configuration, lifecycle API
references/vllm-ascend-model-doc-guide.md — vllm-ascend model adaptation documentation lookup guide: URL structure, model page sections, image tag extraction, deployment form clues, common environment variable quick reference
references/ascend-hardware-matrix.md — Ascend chip model (A2/A3/310P) quick reference: product form, VRAM, card count configuration for different chip × model scales
references/mooncake-pd-deploy-guide.md — Mooncake PD disaggregated deployment guide(PD disaggregation-specific): Mooncake KV transfer, XpYd proxy,--kv-transfer-config/--deployment-config、pd_colocate template modification, single deployment unit dual inference unit PD deployment
references/iam-policies.md — IAM minimum permission policy(required Actions for OBS/SWR/ModelArts/IAM operations and example request bodies for obtaining IAM API tokens)
references/verification-method.md — Deployment verification methods(Method A/B verification steps, OpenAI-compatible invocation checks, common verification failure diagnosis)
references/acceptance-criteria.md — Acceptance criteria(deployment success determination, artifact verification checklist)
references/artifact-check.md — Deployment artifact self-check checklist(mandatory gate before uploading to OBS): hard availability requirements (H01H12) + soft performance requirements (S01S04), against Huawei Cloud best practices + vllm-ascend model documentation
references/troubleshooting.md — Full troubleshooting table(symptom → root cause → troubleshooting action)
references/cli-installation-guide.md — KooCLI (hcloud) Installation Guide: Download address, cross-platform installation steps, AK/SK configuration, common installation issues
scripts/create_infer_service.py — New inference API deployment script (create/list/show/delete/scale;--raw-json overall flat request body;token cannot be passed via command line, provided via IAM_TOKEN/--token-file(600)/interactive)
scripts/check_artifacts.py — Deployment artifact self-check script(mandatory gate at step 3.4): availability hard requirements H01H12 + performance soft requirements S01S04 static validation, FAIL prohibits upload (see checklist at references/artifact-check.md)
scripts/upload_code_to_obs.py — Use OBS Python SDK(esdk-obs-python) to upload deployment artifacts to the OBS code directory (supports --code-obs-path or --model-obs-path inference)
scripts/pull_push_image.sh — Quay.io pull + SWR push helper script (--login pass login command (only supports --password-stdin, password via SWR_PASSWORD environment variable injection)
scripts/get_master_ip.py — multi-node/PD colocation scenariorole identification script(from ModelArts-injected GLOBAL_RANK_TABLE_FILE_PATH global rank table get master IP, dp_rank and --dp-size data parallel total; multi-node/pd_colocate must be copied and uploaded along with the script)
scripts/test_inference.py — OpenAI-compatible API invocation verification script;scripts/download_model.py — weight download command example (for user reference);scripts/obs_upload.sh — obsutil weight upload helper script
templates/start_vllm.single.sh — single-node startup template (single-machine single-card/multi-card TP; built-in weight mount self-check; does not include DEPLOY_FORM runtime branch);templates/start_vllm.multinode.sh — multi-node startup template (master/slave role determination via get_master_ip.py reading rank table,POD_IP comparison;HCCL_IF_IP/GLOO_SOCKET_IFNAME multi-NIC binding;--data-parallel-* parameters;DATA_PARALLEL_SIZE via get_master_ip.py --dp-size dynamically inferred; slave appends --headless)；templates/start_vllm.pd.sh — PD disaggregated startup template (Prefill/Decode dual roles, VLLM_ROLE injection, respective ports);templates/start_vllm.pd_colocate.sh — PD colocated startup template (aligned with official glm_5_2.sh single-engine DP form, no need for VLLM_ROLE/Mooncake; master/slave determined via get_master_ip.py reading rank table;--data-parallel-* parameters + slave --headless; single PORT);templates/health_check.sh — unified health check template (startup/liveness/readiness three types of probes, distinguished as single/multi-node by whether get_master_ip.py exists)

Document source

Huawei Cloud official best practice "Obtaining a model from the vLLM-Ascend open-source community and deploying it on the ModelArts inference platform": https://support.huaweicloud.com/intl/zh-cn/bestpractice-modelarts/modelarts_10_3121.html
Create Inference Service (CreateInferService) API / legacy online service migration / model inference (new version) user guide: https://support.huaweicloud.com/api-modelarts/CreateInferService.html · https://support.huaweicloud.com/intl/zh-cn/inference-modelarts/inference2.0-modelarts-0080.html · https://support.huaweicloud.com/intl/zh-cn/inference-modelarts/inference2.0-modelarts-0002.html
ModelServing combined with Mooncake deployment guide / GLM5.1-w8a8 Atlas800I A2 PD disaggregated deployment (Ascend forum) / ModelArts cross-supernode scheduling troubleshooting: https://support.huaweicloud.com/intl/zh-cn/bestpractice-cce/cce_bestpractice_10082.html · https://www.hiascend.com/dev/forum/thread-02192217146677654095-1-1.html · https://support.huaweicloud.com/intl/zh-cn/trouble-modelarts/inference2.0-modelarts-0042.html