external-cannbot-model-model-infer-superkernel

SuperKernel Adaptation Skill

Provides the complete adaptation process for the SuperKernel operator binary fusion technology, including model analysis, Scope range determination, code implementation, and performance verification.

---

Important Principles

• Prerequisites: Must satisfy exe_mode: "ge_graph", Atlas A3 hardware, PyTorch framework
• Only effective in the Decode phase: Input length changes dynamically in the Prefill phase, and SuperKernel is automatically disabled
• Manually mark Scope: Need to use superkernel_scope context manager to mark the fusion range
• Configuration mutual exclusion: Does not support eager mode and aclgraph mode
• Understand before acting: Determine the Scope range after analyzing the model structure, avoid blindly copying other model configurations
• Combined use: SuperKernel, as one of the optimization technologies, can be used in combination with other optimization technologies (multi-stream parallelism, fused operators, etc.)

---

Important Reminder

All four steps must be completed：

1. Analyze model structure
2. Determine Scope range
3. Code writing
4. Performance testing and verification(must be executed, cannot be skipped)

Final Deliverables：

• Optimized code files (config.yaml, modeling_*.py)
• SuperKernel optimization document (superkernel_optimization_report.md）

---

Workflow

Step 1: Analyze model structure

Goal: Understand the model architecture and identify the parts where SuperKernel can be applied

1. Read model code：

   • Find the model implementation file (cann-recipes-infer/models/{model_name}/models/modeling_*.py）
   • Find the configuration file (cann-recipes-infer/models/{model_name}/config.yaml）
   • Find the runner file (cann-recipes-infer/models/{model_name}/models/runner_*.py）

2. Identify model architecture features：

   • Whether there are Attention layers (Self-Attention, Cross-Attention)
   • Whether there are MoE layers (expert routing, expert computation)
   • Whether there are other computation-intensive modules (FFN, LayerNorm, etc.)
   • Whether Prefill and Decode are separated

3. Check the current configuration：

   • exe_mode Whether it is ge_graph(required)
   • enable_superkernel Current status
   • Whether the hardware is Atlas A3
   • Whether graph mode is enabled (torch.compile）

4. Identify potential issues：

   • Dynamic shape issues (whether KV Cache is preallocated)
   • Control flow issues (whether there are Python if/for)
   • Unsupported operators (Tiling sink operators)

Deliverables：

• Model structure analysis (internal record, no file output)
• List of modules where SuperKernel can be applied
• List of potential issues

---

Step 2: Determine Scope range

Goal: Determine the fusion range of SuperKernel

Use AskUserQuestion to provide the following options:

Option	Description	Applicable scenarios	Pros and cons
Attention module only	Only include Attention computation in the SuperKernel scope	First attempt, model bottleneck is in Attention	Low risk, easy to debug
MoE module only	Only include MoE expert computation in the SuperKernel scope	MoE architecture models	Highly targeted, obvious effect
Full-model Decoder layer	Include the entire Decoder layer in the SuperKernel scope	Simple model structure, existing successful cases	Largest fusion scope, highest performance improvement potential, greater debugging difficulty
Use automatic analysis tool	Launch a subagent to analyze the optimal Scope partition	Pursue optimal performance	Automatic recommendation based on performance analysis, takes a long time (5-10 minutes)
Custom Scope	Manually specify the modules to be fused	Have special requirements or already know the optimal configuration	High flexibility

Execution content：

• Generate the corresponding Scope configuration according to the user's choice
• If "automatic analysis tool" is selected, launch a subagent to perform performance analysis
• Generate Scope marker code snippets
• Determine the files and specific locations that need to be modified

Output：

• Scope configuration plan (internal record, no file output)
• Code locations that need to be modified and the specific changes

For detailed Scope analysis methods, see references/scope-analysis-guide.md

---

Step 3: Code writing

Goal: Implement SuperKernel adaptation code

Important: This step only modifies code and does not output any intermediate documents.

3.1 Configuration file modification

Modify cann-recipes-infer/models/{model_name}/config.yaml：

# 确保以下配置
exe_mode: "ge_graph"              # 必须是 ge_graph
model_config:
  enable_superkernel: True        # 启用 SuperKernel
  enable_multi_streams: False     # 根据需求配置
  enable_cache_compile: False     # 根据需求配置

3.2 Model code modification

In cann-recipes-infer/models/{model_name}/models/modeling_*.py :

# 1. 导入上下文管理器
from executor.utils import superkernel_scope

# 2. 在 decode 方法中添加 Scope 标记
class YourModel:
    def decode(self, input_ids, ...):
        is_prefill = False  # decode 阶段

        # 根据选择的 Scope 策略添加
        with superkernel_scope(
            self.enable_superkernel and not is_prefill,
            label="decode_layers",  # 或 "attention_only", "moe_only"
            option="stream-fusion=1"  # 编译选项
        ):
            for decoder_layer in self.layers:
                # 运算逻辑
                hidden_states = decoder_layer(
                    hidden_states,
                    attention_mask=attention_mask,
                    ...
                )

        return hidden_states

Example Scope marker locations：

• Attention only: In the forward method of the Attention module
• MoE only: In the forward method of the MoE layer
• Full Decoder layer: Outside the entire Decoder layer loop

Output：

• Modified code files (config.yaml, modeling_*.py, etc.)

---

Step 4: Performance testing verification (must be executed)

Goal: Verify the correctness and performance improvement of SuperKernel adaptation

Important: This step must be executed and cannot be skipped. Perform performance testing immediately after completing code modifications.

4.1 Establish performance baseline

1. Temporarily disable SuperKernel：

   cd cann-recipes-infer/models/{model_name}
   # 备份当前配置
   cp config.yaml config.yaml.superkernel
   # 禁用 SuperKernel
   sed -i 's/enable_superkernel: True/enable_superkernel: False/' config.yaml
   

2. Run baseline test：

   bash infer.sh 2>&1 | tee baseline.log
   

3. Record baseline performance：

   • Decode single-step latency (ms)
   • Throughput (tokens/s)
   • Extract key performance metrics from logs

4.2 Test the optimized version

1. Restore SuperKernel configuration：

   # 恢复 SuperKernel 配置
   cp config.yaml.superkernel config.yaml
   

2. Run optimization test：

   bash infer.sh 2>&1 | tee optimized.log
   

3. Record optimized performance：

   • Decode single-step latency (ms)
   • Throughput (tokens/s)
   • Extract key performance metrics from logs

4.3 Performance comparison analysis

Calculate performance improvement:

性能提升 = (基线耗时 - 优化后耗时) / 基线耗时 × 100%
吞吐量提升 = (优化后吞吐量 - 基线吞吐量) / 基线吞吐量 × 100%

4.4 Generate optimization report

Must generate: Create superkernel_optimization_report.md, containing:

# SuperKernel 优化报告

## 优化配置

- 模型：{model_name}
- Scope 策略：{scope_strategy}
- 优化日期：{date}

## 代码修改

### 配置文件修改
- 文件：`config.yaml`
- 修改内容：启用 `enable_superkernel: True`

### 模型代码修改
- 文件：`cann-recipes-infer/models/modeling_*.py`
- 修改内容：添加 SuperKernel scope 标记
- Scope 范围：{scope_description}

## 性能测试结果

### 基线性能（SuperKernel 禁用）
- Decode 单步耗时：{baseline_latency} ms
- 吞吐量：{baseline_throughput} tokens/s

### 优化后性能（SuperKernel 启用）
- Decode 单步耗时：{optimized_latency} ms
- 吞吐量：{optimized_throughput} tokens/s

### 性能提升
- 延迟降低：{latency_improvement}%
- 吞吐量提升：{throughput_improvement}%

## 结论

{conclusion}

## 使用说明

优化后的模型配置已保存，可直接使用：
\`\`\`bash
cd cann-recipes-infer/models/{model_name}
bash infer.sh
\`\`\`

Outputs：

• Performance test logs (baseline.log, optimized.log)
• SuperKernel optimization report（superkernel_optimization_report.md）

For detailed performance baseline establishment methods, see references/performance-baseline-guide.md

---

Final deliverables

After completing all four steps, you must provide:

1. Optimized code files：

   • cann-recipes-infer/models/{model_name}/config.yaml(SuperKernel enabled)
   • cann-recipes-infer/models/{model_name}/models/modeling_*.py(Scope markers added)

2. SuperKernel optimization report：

   • superkernel_optimization_report.md
   • Contains complete performance comparison data and optimization description

No need to output：

• Intermediate analysis document (superkernel_analysis.md)
• Scope planning document (superkernel_scope_plan.md)
• Other temporary files

---

Scope selection guide

Quick decision tree

模型类型？
├─ MoE 模型 → 优先选择"仅 MoE 模块"
├─ 标准 Transformer → 优先选择"仅 Attention 模块"
└─ 简单模型 + 有成功案例 → 可选择"全 Decoder 层"

首次尝试？
└─ 是 → 选择"仅 Attention 模块"（风险最小）

追求最优性能？
└─ 是 → 选择"使用自动分析工具"

已知最优配置？
└─ 是 → 选择"自定义 Scope"

Scope templates

For detailed Scope templates, see the resources/scope-templates/ directory:

• attention-only.md - Attention-only Scope template
• moe-only.md - MoE-only Scope template
• full-model.md - Full-model Scope template

---

Configuration checklist

Before enabling SuperKernel, please ensure:

• exe_mode is set to ge_graph(cannot be eager or aclgraph）
• Hardware is Atlas A3 series
• Framework is PyTorch
• The model has distinguished prefill and decode phases
• SuperKernel scope has been correctly marked

---

Reference documents

Internal documents

• references/scope-analysis-guide.md - Detailed Scope analysis guide
• references/performance-baseline-guide.md - Performance baseline establishment guide
• resources/scope-templates/ - Scope templates

External documents

• Official SuperKernel development documentation
• PyTorch graph mode user guide
• DeepSeek-R1 decode performance optimization
• SuperKernel usability guide

---

Best practices

1. First attempt: Choose "Attention module only", lowest risk
2. Gradually expand: Expand the Scope only after successful verification
3. Performance comparison: Perform a performance comparison after every modification
4. Documentation: Record the configuration and results of each attempt
5. Troubleshooting: When encountering problems, first check the checklist and common issues
6. Seek help: For complex problems, you can invoke other skills or consult detailed documentation