external-cannbot-model-model-infer-multi-stream

NPU Multi-Stream Whole-Network Optimization Skill

For the NPU multi-stream whole-network optimization task in this repository, proceed in the order of "analysis → development and debugging → acceptance". Prioritize consulting existing cases, code patterns, and the official API index in the repository.

Overview

This skill addresseswhole-network multi-stream optimization。

The main workflow is fixed as follows:

1. First break down the whole network into modules, draw the module-level DAG, and determinewhich modulescan run in parallel with each other.
2. Then foreach module, break down the operators, draw the intra-module operator DAG, and determine which operators within the module can run in parallel.
3. Only after the dependencies are clear should multi-stream development, debugging, and acceptance be carried out.

The analysis and implementation results are uniformly recorded in:

multi-stream-analysis/<network_or_case_name>.md

The report structure uses references/report-template.md。

Important Principles

• Whole network before local: first answer the whole-network main path and inter-module parallelism, then go into intra-module operators.
• Clear module-level goal: the goal of module decomposition is to determinewhich modulescan run in parallel with each other.
• Every module must be drilled down into: the second layer must complete the operator list, operator DAG, and intra-module parallelism conclusion for every module.
• Synchronization must be explicit: multi-stream transformation must have clear event / wait / wait_stream / wait_tensor relationships.
• The switch must be able to be turned off: the multi-stream path must retain an enable switch and the original fallback path, to facilitate debugging and acceptance.
• Prove correctness first, then pursue performance: first verify dependencies, functionality, and accuracy, then look at overlap and latency gains.
• Overlap does not equal gain: when tailing, resource contention, host bound, or shape degradation occurs, continue to evaluate core control and graph mode limitations.
• Do not mix and copy cases: you must first determine the execution mode of the current model, then choose one main API path; do not mix and apply eager and graph styles together.
• Optimal orchestration for debugging and optimization: the split-flow orchestration of optimization points is not unique; during debugging, try different orchestrations and choose the best-performing solution.
• For official constraints, prioritize upstream docs/zh: first read the corresponding path's multi_stream.md; only when overlap is correct but resource contention, tailing, or deadlock risk occurs, then read the same path's limit_cores.md。
• Development and debugging interleave: development and debugging proceed in an interleaved manner, paying attention to whether operators/modules are on the corresponding streams, and the cube/vector utilization (number of cores/occupancy time).
• Use profiler agent for debugging: when a profiler agent exists, use the profiler agent to confirm performance gains and resource usage. When no profiler agent exists, add prints yourself to make judgments.

Execution Path Delimitation

Before starting, first confirm which multi-stream path the current code takes:

• torchair.CompilerConfig() + torchair.get_npu_backend() + GE graph mode: take the Ascend IR path. First look at cann-recipes-infer/docs/zh/ascend_ir/features/advanced/multi_stream.md; when core control is needed, then look at cann-recipes-infer/docs/zh/ascend_ir/features/advanced/limit_cores.md。
• torch.compile(..., backend="npugraph_ex"): take the npugraph_ex / aclgraph path. First look at cann-recipes-infer/docs/zh/npugraph_ex/advanced/multi_stream.md, and only when core control is needed look at cann-recipes-infer/docs/zh/npugraph_ex/advanced/limit_cores.md。
• Do not mix the in-graph API of Ascend IR with the explicit stream API of aclgraph in the same implementation.
• GE graph mode is usually delimited by CompilerConfig and get_npu_backend() ; do not treat torch.compile(mode=...) as the current main entry-point style.
• npugraph_ex The path is essentially explicit stream / event / lifetime management; if the code is already in this style, prefer to continue along the existing implementation instead of forcibly changing it to TorchAir in-graph expression.

Workflow

Step 1: Analyze

profiler agent When a profiler agent exists, the following information can be obtained through the profiler agent:

• Propose a module decomposition plan, and obtain from the profiler agent the operator sequences within the module and the corresponding code snippets

1. First read references/report-template.md, and under cann-recipes-infer/docs/common/multi-stream-analysis/<model_name> create or update the result file.
2. Follow references/module-decomposition-spec.md to complete whole-network module decomposition:
   • Determine whether the analysis phase is prefill or decode
   • Produce the whole-network module list, dependency list, and Mermaid module DAG
   • Output the module-level parallelism conclusion, consistently using "main serial chain / parallelizable group / to-be-verified group / recommended stream grouping"
3. Foreach modulecontinue with operator-level decomposition:
   • Produce the operator list within the module, dependency list, and Mermaid operator DAG
   • Output the parallelism conclusion within the module, still using grouped descriptions
4. You can refer to cases based on the model structure. When selecting a case, first read examples/README.md, and use its "Quick Selection Table" to enter the closest pattern.

Step 2: Develop

1. First determine the adaptation path based on the currently used graph mode backend:
   • Use torchair.CompilerConfig() + torchair.get_npu_backend(): take the Ascend IR / GE graph mode path
   • Use torch.compile(..., backend="npugraph_ex"): take the npugraph_ex / aclgraph path
2. Enable multi-stream expression:
   • Ascend IR / GE graph mode: prefer npu_stream_switch
   • npugraph_ex / aclgraph: prefer torch.npu.Stream() + torch.npu.stream()
3. Add timing control:
   • Ascend IR / GE graph mode: prefer npu_wait_tensor; when there is an existing tagged event style, continue using npu_tagged_event_*
   • npugraph_ex / aclgraph: prefer torch.npu.Event()、record_event()、wait_event()
4. The explicit stream path requires separately checking memory lifetime:
   • If a short-lived tensor will continue to be consumed by other streams, add record_stream()
   • Long-lived objects such as model weights and resident caches generally do not need to be added
5. Prefer to dominimal verifiable overlap：
   • Change only one clear parallelism point
   • First add synchronization, then expand the parallelism window
   • First keep the original path and enable switch
6. If overlap has been achieved but there is still obvious tailing, then evaluate:
   • Ascend IR / GE graph mode:limit_core_num or global config.ge_config.aicore_num
   • npugraph_ex / aclgraph：torch.npu.npugraph_ex.scope.limit_core_num
   • set_stream_limit / get_stream_limit
7. Before amplifying changes, do a minimal verification first:
   • Compare single-stream and multi-stream baselines
   • Use the profiler to confirm whether overlap actually occurs, rather than logically splitting streams while execution is still serial
8. Write all development decisions back to the result md: clearly describe stream grouping, synchronization points, switches, fallback paths, and expected benefits.

Step 3: Debug

profiler agent When a profiler agent is present, the following information can be obtained through the profiler agent:

• Obtain the operator sequence and the corresponding execution time of the operators for a certain module
• Obtain detailed runtime information of a certain operator (number of cube/vector cores occupied, MTE2/MTE3/Cube/Vector execution time, etc.)
• Obtain the runtime status of each stream for a certain module
• Obtain the execution time of a certain module Ensure that definite information is obtained from profile data in a timely manner during debugging, and do not rely on guessing.

If there is more than one possible multi-stream orchestration method, try different orchestration methods and choose the solution with the best performance. During debugging, handle the following 4 categories of problems separately, and do not investigate them all mixed together:

1. Dependency errors / synchronization errors
   • Focus on checking event records, wait order, cross-stream convergence points, and the order of shared state writes
   • Typical symptoms: reading incomplete results, waiting forever, intermittent incorrect results
2. Precision or functional abnormalities
   • First compare with the pre-optimization baseline, then follow prefill/decode → 模块 → 算子 to narrow down the scope
   • Determine whether it is a state timing issue introduced by multi-stream, rather than a problem with the operator itself
3. No performance gain
   • Focus on shape changes, increased task count, host bound, bandwidth contention, resource contention between streams, and tailing
   • First determine whether Cube resources are already basically fully utilized, and then decide whether to continue evaluating core control or narrowing the overlap range
4. Graph mode / runtime limitations
   • Focus on graph break, graph mode API constraints, stream semantic differences, and runtime unsupported features
   • For graph scenarios, prioritize troubleshooting along the TorchAir path, and do not fall back to forcing an eager mindset
5. aclgraph lifecycle errors
   • Focus on checking whether short-lived tensors continue to be used across streams, and whether there is any omission of record_stream()
   • Typical symptoms: intermittent incorrect results, data corruption, segmentation faults, or abnormal memory issues

Step 4: Acceptance

Acceptance must cover at least the following 4 categories, and the results must be written back to the report:

1. Functional acceptance
   • The output is consistent before and after enabling multi-stream
   • After the switch is turned off, the original path can run normally
2. Synchronization acceptance
   • There are no missing waits before and after the convergence point
   • There is no read/write reordering for shared state, KVCache, or communication results
3. Performance acceptance
   • Record the key latency, throughput, or single-step execution time before and after optimization, and confirm that there is indeed overlap and latency improvement
   • Confirm that the execution time of operators or modules outside the optimization points has not degraded
   • If there is no gain, it must be clarified whether it is an implementation problem or the scenario is unsuitable
4. Profile acceptance
   • Confirm that overlap indeed exists, rather than being logically divided into streams but still executed serially in practice
   • Confirm whether the key tailing has been shortened, and whether new gaps or resource contention have appeared

API selection rules

eager / patch style

Preferred use:

• torch.npu.Stream()
• record_event()
• wait_event()
• wait_stream()

Applicable scenarios:

• patch-form transformation
• The existing code already explicitly uses torch.npu.current_stream()
• The runtime does not use multi-stream expression within the TorchAir graph

graph / TorchAir style

Preferred use:

• npu_stream_switch
• npu_wait_tensor
• npu_record_tagged_stream
• npu_tagged_event_wait

Applicable scenarios:

• ge_graph / TorchAir multi-stream expression
• The code already explicitly uses the scope / tagged event style
• The synchronization relationships need to be expressed within the graph

Tail effect and resource issues

When "overlap is present but the benefit is unstable" occurs, then evaluate:

• limit_core_num
• torch_npu.set_stream_limit
• torch_npu.get_stream_limit

Order of use:

1. First confirm that dependencies and overlap are correct
2. Then check whether tail effect or resource contention exists
3. Finally introduce core control, stream limit, or prefetching

For detailed routing, see references/api-routing.md。

Implementation checklist

• Confirmed whether the current implementation goes through Ascend IR / GE or npugraph_ex
• Confirmed that truly parallelizable branches exist, rather than forcibly splitting a serial chain into multiple streams
• Cross-stream dependencies have been explicitly expressed, rather than relying on implicit synchronization
• In the explicit stream path, cross-stream use of short-lived tensors has been checked record_stream()
• Only after overlap is already achieved, continue evaluating core control, stream limit, or prefetching
• Confirmed through baseline and profiler that the optimization direction holds

Reference documentation index

Read on demand.

• Whole-network module decomposition specification： references/module-decomposition-spec.md
• Documentation and API routing： references/api-routing.md
• Case main entry and selection table： examples/README.md
• Fixed report template： references/report-template.md