msmodelslim-quant

msmodelslim-quant

Huawei msmodelslim quantization tool skill - dynamic query + containerized deployment

Skill Description

This skill is specifically customized for Huawei Ascend NPU servers, and is used for LLM quantization acceleration on Ascend NPU cards. msmodelslim is an open-source model compression tool in the Huawei Ascend ecosystem, supporting INT4/INT8/INT16 quantization of large models on Ascend NPUs, significantly reducing model memory usage and improving inference performance. This skill provides a complete quantization workflow, including environment checks, dynamic solution query, container deployment, quantization execution, etc.

Applicable Scenarios：

• Perform model quantization on Ascend NPU servers
• Deploy quantized large models to Ascend NPUs for inference
• Optimize model performance and reduce memory usage

Supported Quantization Types：W4A8, W4A8C8, W8A16, W8A8, W8A8S, W8A8C8, W16A16S

Keywords

msmodelslim, quant, model quantization, 华为昇腾, Ascend, NPU, 
模型compress, quant加速, INT4quant, INT8quant, 大模型deployment,
昇腾NPU, 华为AI, Atlas, Davinci, 模型加速, 权重量化,
ascend, 昇腾卡, npu-smi, 模型deployment, LLMquant

Trigger Conditions

Triggered when the user asks about the following:

• "quantize XXX model"
• "XXX model quantization"
• "msmodelslim supports XXX"
• "XXX quant"
• Use msmodelslim for model quantization

Core Principles

Do not hardcode the model support list - always dynamically query the actual project to determine available features

Remote Server Access

⚠️ Important Note: If the user's instruction is to perform quantization operations on aremote server, then prioritize usingnon-interactivetools to access the server, for example:

• sshpass -p 'password' ssh user@server
• sshpass -f passwordfile ssh user@server
• Configure SSH Key and use ssh -i keyfile user@server

Avoid directly using the ssh user@server command, because this will lead to an interactive session, which may cause:

• Inability to automate execution of quantization scripts
• Difficulty in model file transfer
• Complicated interaction with the remote server

It is recommended to complete all preparatory work locally (model download, configuration checks, etc.), and then transfer files to the server in batches via scp or rsync .

Model Naming Convention

Naming format for quantized models:{original_name}-{quant_type}-{features}

Examples:

• qwen2-7b-w8a8 (Qwen2-7B, W8A8 quantization)
• qwen3-32b-w8a8s-QuaRot (Qwen3-32B, W8A8 sparse quantization, QuaRot features)
• deepseek-v3-w4a8c8 (DeepSeek-V3, W4A8C8 quantization)

Ascend NPU Server Environment

This skill is specifically customized for Huawei Ascend NPU servers. Before starting quantization, please ensure you understand the following information:

View Ascend NPU Card Information

@scripts/npu-commands.sh

Ascend NPU Model Reference

@references/npu-models.md

Huawei Ascend Ecosystem Components

@references/ecosystem.md

Ascend NPU Quantization Pre-checks

@scripts/env-check.sh

Workflow

Step 1: Information Extraction and Path Confirmation

Extract the following information from the user's description:

• Working directory (work_path): The location where project files and scripts are stored
• Original model path (weights_path): The location of the original model weight files
• Model name (model_name): The name of the model to be quantized
• Quantization format (quant_format): Such as W8A8, W4A8, W8A8S, etc.
• Features (features): optional, such as QuaRot, AWQ, etc.

Must be confirmed with the user：

1. Working directory (work_path)
2. Save path (save_path) - the output location of the quantized model

Step 2: Check/create the msmodelslim project

@scripts/msmodelslim-setup.sh

Step 3: Dynamic query quantization solution

@scripts/quant-queries.sh

Step 4: Containerized deployment (optional)

Ask the user whether a container is needed. If so:

@scripts/docker-commands.sh

Step 5: Check model-specific environment requirements

Important: After installing msmodelslim, you must check whether the model has special environment requirements. These requirements are usually documented in example/{model_name}/README.md .

Important: During the process of installing dependencies with pip, if a pip installation timeout occurs, you can ask the user whether to change the pip source. You can choose a preset pip source:pip config set global.index-url http://mirrors.aliyun.com/pypi/simple && pip config set install.trusted-host mirrors.aliyun.com, or let the customer enter a specific suitable source. This speeds up the installation of environment dependencies.

5.1 Check the model-specific README

@scripts/env-check.sh

5.2 Check the environment requirements in the README

@references/quant-types.md

5.3 Common environment check items

@scripts/env-check.sh

5.4 Apply environment changes

@scripts/env-check.sh

5.5 Update the pre-quantization report

@templates/quant-report-template.md

Step 6: Pre-quantization report

@templates/quant-report-template.md

⚠️ Step 6.5: User interactive confirmation (required)

Before executing quant, you must perform interactive confirmation with the user：

1. Show the complete pre-quantization report to the user
2. Clearly inform the user of the following information:
   • quant command and all parameters
   • Estimated time
   • Output directory location
   • Possible risks
3. Ask the user:"Is the report information correct? Can the quantization solution be executed?"
4. Only after the user explicitly confirms, can you proceed to Step 7 to execute quant
5. If the user has any questions or requests modifications, you must stop to answer or adjust the solution

⚠️ It is strictly prohibited to skip this confirmation step and directly execute quant; you can continue only after obtaining an explicit answer from the user.

Step 7: Execute quant

After the user confirms, execute the quant command:

@scripts/quant-execute.sh

Step 8: Completion summary

After quant is completed, return a summary:

@templates/quant-report-template.md

quant type reference

@references/quant-types.md

Dynamic query command reference

Model groups supported by query

quant types supported by query

query model adapters

@scripts/quant-queries.sh

Official resources

@references/official-resources.md

Notes

1. Dynamic query first: Do not assume whether a model is supported or not; you must actually query the project
2. Path confirmation: You must confirm work_path and save_path with the user before executing
3. Containerization recommendation: Containers are recommended for production environments to ensure environment consistency
4. Version compatibility: Different versions of msmodelslim may support different models and quant types
5. Hardware requirements: Some quant operations may require specific NPU or GPU hardware
6. Pre-quantization report: A report must be generated before execution for user confirmation