mindspeed-llm-train-profiler

MindSpeed-LLM Training Profiling Data Collection (Ascend NPU)

• Reply in the user's language.
• Do not translate the following: commands, file paths, environment variables, package names, error messages.

Step 0: Confirm prerequisites

Reply to the user:

Before starting Profiling collection, please provide the following information:

Required:

• Training script path (.sh) — a runnable script with the model, data, and weights already configured

Optional (with default values):

• Collection level: level0 / level1(recommended) / level2
• Collection steps: default step 2 ~ 4 (excluding step 4; actually collects step 2 and 3)
• Collection content:CPU(enabled by default), memory, stack, tensor shape
• Number of NPU cards: single card by default

Without a training script, collection cannot be performed.

When the user does not provide any information: Stop. Reply: "Profiling requires a runnable training script. Please prepare it before collecting."

After the user provides the training script: Display the final configuration table and wait for the user to confirm before executing.

Configuration item	Value
Model	(read from the script)
Training script	(path)
Number of NPU cards	(read NPUS_PER_NODE from the script)
Collection level	level1 (recommended)
Collection steps	step 2 ~ 4 (excluding 4)
Collection card number	rank 0
CPU collection	Enabled
Memory collection	Disabled
Stack collection	Disabled
Tensor shape collection	Disabled

Start after confirmation. If modifications are needed, please specify.

Do not download models, convert weights, or create training scripts.

User input and variable mapping

User says	Variable	Default value
Training script (.sh)	TRAINING_SCRIPT	(required)
"CPU" / "collect CPU"	PROFILE_WITH_CPU=true	true
"memory" / "memory"	PROFILE_WITH_MEMORY=true	false
"Level0/1/2"	PROFILE_LEVEL	level1
"step N to M"	PROFILE_STEP_START=N PROFILE_STEP_END=M	2, 4
"stack" / "stack"	PROFILE_WITH_STACK=true	false
"shape" / "dimensions"	PROFILE_RECORD_SHAPES=true	false
"all cards" / "all ranks"	PROFILE_RANKS=-1	0(only rank 0)

Note: The number of NPU cards is read from the training script (NPUS_PER_NODE or --nproc_per_node). Profiling collects only rank 0 by default. Set PROFILE_RANKS=-1 to collect all cards only when the user explicitly requests it.

Step 1: Check the environment

Run npu-smi info to confirm that the NPU is available. If it is not available,stop: "No NPU detected in the current environment. Please run in an environment with Ascend NPU."

To load the CANN environment:source /usr/local/Ascend/ascend-toolkit/set_env.sh

Step 2: Verify the script and run Profiling

Do not modify the user's original training script. Create a new script based on the original script.

1. Read the original script and check whether the key paths it references exist (weight directory, data files, tokenizer directory).
2. If any path does not exist,stop and report:

   A path referenced in the training script does not exist:

   • <缺失路径> — not found

   Please check the path configuration in the script and ensure the training environment is ready before performing Profiling collection.

3. Createa new Profiling script (use a timestamped filename, e.g. run_profiling_$(date +%Y%m%d%H%M%S).sh, to avoid overwriting an existing file). Copy the contents of the original script and add the following Profiling parameters to the torchrun command (before | tee ; if there is no | tee , append to the end of the torchrun command; if the command format cannot be recognized, show the parameters and ask the user to specify the insertion position):

--profile --profile-step-start 2 --profile-step-end 4 \
--profile-ranks 0 --profile-level level1 --profile-with-cpu \
--profile-save-path ./profiling_output

1. If the original script already contains the --profile parameter, note the existing configuration and ask the user whether to keep or override it.
2. Adjust parameter values according to the user's Profiling configuration. Create the output directory before running.
3. Runthe new script. The original script must remain unchanged.
4. If training fails (non-zero exit code), report the error to the user:

   Training script execution failed (exit code: N). Profiling collection requires training to run normally.

   Please fix the training script and retry.

Step 3: Verify and report the results

After training completes, verify the Profiling output:

1. Check whether the Profiling output directory exists and whether it contains the *_ascend_pt/ subdirectory.
2. It should contain at least PROF_*/device_*/data/(NPU data).
3. Report to the user: output location, configuration used, directory structure.
4. It is recommended to use MindStudio Insight for visual analysis.

If the output is missing or empty, check:PROFILE_STEP_START >= 1, whether training reached the collection step, and whether the CANN environment has been loaded.

Troubleshooting

Symptom	Solution
NPU out of memory	Reduce the batch size in the training script
Profiling directory is empty	PROFILE_STEP_START must be >= 1; training must reach this step
Address already in use	Modify the MASTER_PORT
Weight shape mismatch	The weight's TP/PP is inconsistent with the training configuration; stop collection after notifying the user

Prohibitions

• Do not modify the user's original training script; a new script must be created.
• Do not install frameworks, download models, or convert weights.
• Do not set --profile-step-start 0 — must be >= 1.
• Do not collect all steps in large-scale training — 2-3 steps is sufficient.
• Do not continue execution without a training script.
• Unless the user explicitly requests it, do not set --profile-ranks -1。

Reference documentation

See reference/mindspeed-profiling-args.md for the complete parameter table (Mcore CLI parameters + FSDP2 YAML configuration).