Launch and monitor model training on Ascend NPU environment.
When to Invoke
- User wants to start model training
- User wants to run performance training
- User wants to run accuracy training
- User asks about training options
Information to Collect
Ask user for the following:
1. Training mode preference:
- Performance training (FP32/FP16)
- Accuracy training (full epochs)
- Custom training
2. Number of GPUs/NPUs to use
3. Batch size (if custom)
4. Any specific training parameters
Training Options
Provide user with training options:
Training Options:
1. Performance Training (FP32, 8 GPUs) - Quick performance test
2. Performance Training (FP16, 8 GPUs) - Quick performance test with mixed precision
3. Accuracy Training (24 epochs) - Full accuracy training
4. Custom training configuration
Workflow
Step 1: Select Training Mode
Ask user to select training mode or provide custom configuration.
Step 2: Prepare Environment
Ensure environment variables are set:
export ASCEND_SLOG_PRINT_TO_STDOUT=0
export ASCEND_GLOBAL_LOG_LEVEL=3
export TASK_QUEUE_ENABLE=2
export COMBINED_ENABLE=1
export HCCL_WHITELIST_DISABLE=1
export HCCL_IF_IP=$(hostname -I | awk '{print $1}')
export HCCL_CONNECT_TIMEOUT=1200
Step 3: Launch Training
Execute training script based on selected mode.
Step 4: Monitor Progress
Check training log and verify training started successfully.
BEVFormer Training Commands
Performance Training (FP32, 8 GPUs)
cd <working_directory>
bash test/train_performance_8p_base_fp32.sh --batch-size=1 --num-npu=8
Performance Training (FP16, 8 GPUs)
cd <working_directory>
bash test/train_performance_8p_base_fp16.sh --batch-size=1 --num-npu=8
Accuracy Training (24 epochs)
cd <working_directory>
bash test/train_full_8p.sh --batch-size=1
Custom Training
cd <model_directory>
bash ./tools/dist_train.sh ./projects/configs/bevformer/bevformer_base.py <num_gpus>
Training Verification
Check Training Started
# Check training process
ps aux | grep torchrun
# Check training log
tail -20 <working_directory>/test/output/train_performance_8p_base_fp32.log
Success Indicators
Training is considered successfully started when:
- Log file shows training iterations
- Loss values are being printed
- No error messages in recent log
- Process is running (torchrun)
Example successful log:
2026-03-12 09:47:40,838 - mmdet - INFO - Epoch [1][26/41] lr: 7.333e-05, eta: 0:36:35, time: 1.883, data_time: 0.023, memory: 25332, loss_cls: 1.2614, loss_bbox: 1.7827
Monitoring Training
Check Training Log
tail -f <working_directory>/test/output/train_performance_8p_base_fp32.log
Key Metrics
- Loss values: loss_cls, loss_bbox, total loss
- Learning rate: lr
- Time per iteration: time
- Memory usage: memory
- ETA: estimated time remaining
Validation Metrics (per epoch)
- NDS: NuScenes Detection Score
- mAP: mean Average Precision
- mATE: mean Translation Error
- mASE: mean Scale Error
- mAOE: mean Orientation Error
Training Completion
Training is complete when:
- All epochs finished
- Final validation metrics printed
- Checkpoint saved
Check for completion:
grep "Saving checkpoint" <log_file> | tail -1
Troubleshooting
Training Fails to Start
- Check NPU devices:
npu-smi info - Check environment variables
- Check conda environment is activated
- Check dataset and weights are linked
Out of Memory
- Reduce batch size
- Reduce number of GPUs
- Enable gradient checkpointing
Training Hangs
- Check NPU status:
npu-smi info - Check for dead processes:
ps aux | grep python - Check log for last activity
Port Already in Use
# Kill existing training processes
pkill -f torchrun
pkill -f dist_train
Log File Locations
| Training Mode | Log File |
|---|---|
| FP32 Performance | test/output/train_performance_8p_base_fp32.log |
| FP16 Performance | test/output/train_performance_8p_base_fp16.log |
| Accuracy | test/output/train_full_8p.log |
Reference
- Training scripts:
<model_directory>/test/ - Model config:
<model_directory>/projects/configs/ - DrivingSDK README:
DrivingSDK/model_examples/<model>/README.md