diffusers-ascend-pipeline

Diffusers Ascend NPU Pipeline Inference

This Skill guides users to run HuggingFace Diffusers Pipeline inference on Huawei Ascend NPU, applicable to any Diffusers model.

Usage Order

Use this Skill in the following order:

1. First perform a pre-check to confirm that the environment, NPU, and weights are available.
2. Then run image or video Pipeline inference.
3. Enable memory optimization, LoRA, or multi-card inference when needed.
4. When querying the API, first locate the GitHub API path by version, then open the official website documentation.

Prerequisites

Dependency	Description	Reference
CANN + torch_npu	NPU runtime environment	diffusers-ascend-env-setup
Model weights	Real weights or fake weights	diffusers-ascend-weight-prep
diffusers	pip install diffusers["torch"] transformers accelerate

Quick Start

1. Pre-check

Run the pre-check script to confirm that the environment, NPU memory, and model weights are ready:

python scripts/validate_pipeline.py --model ./my_model --device npu:0 --min-memory 16

Check items:

Check Item	Description
Python packages	torch, torch_npu, diffusers, transformers
CANN environment	Environment variables such as ASCEND_HOME_PATH
NPU availability	Number and names of devices
NPU memory	Whether free memory meets the minimum requirement
Model weights	model_index.json exists and components are complete

2. Image Model Inference

General mode(applicable to any image Pipeline such as FLUX, SDXL, SD3, etc.):

import torch
import torch_npu

from diffusers import DiffusionPipeline

# 加载 Pipeline（自动识别 Pipeline 类型）
pipe = DiffusionPipeline.from_pretrained(
    "./my_model",
    torch_dtype=torch.bfloat16,
)
pipe = pipe.to("npu:0")

# 生成图像
generator = torch.Generator("npu").manual_seed(42)
image = pipe(
    prompt="a cat sitting on a windowsill, watercolor style",
    num_inference_steps=20,
    guidance_scale=3.5,
    generator=generator,
).images[0]

image.save("output.png")

Use script:

# FLUX.1-dev 推理
python scripts/run_pipeline.py \
    --model ./fake_flux_dev \
    --prompt "a cat sitting on a windowsill" \
    --device npu:0 --dtype bfloat16 \
    --steps 20 --seed 42 \
    --output flux_output.png --benchmark

# SDXL 推理
python scripts/run_pipeline.py \
    --model stabilityai/stable-diffusion-xl-base-1.0 \
    --prompt "a beautiful landscape" \
    --device npu:0 --dtype float16 \
    --steps 30 --guidance-scale 7.5 \
    --output sdxl_output.png --benchmark

3. Video Model Inference

Video Pipelines (Wan, CogVideoX, etc.) output frame sequences, and the script automatically detects and exports them as MP4:

import torch
import torch_npu

from diffusers import DiffusionPipeline
from diffusers.utils import export_to_video

pipe = DiffusionPipeline.from_pretrained(
    "./wan_model",
    torch_dtype=torch.bfloat16,
)
pipe = pipe.to("npu:0")

generator = torch.Generator("npu").manual_seed(42)
output = pipe(
    prompt="a dog running on the beach",
    num_inference_steps=30,
    generator=generator,
)

# 导出视频
export_to_video(output.frames[0], "output.mp4", fps=16)

Use script:

python scripts/run_pipeline.py \
    --model ./wan_model \
    --prompt "a dog running on the beach" \
    --device npu:0 --dtype bfloat16 \
    --steps 30 --seed 42 \
    --output wan_output.mp4 --benchmark

4. Memory Optimization

Large model inference may encounter insufficient NPU memory; the following optimization techniques can be used in combination:

Technique	Method	NPU Compatibility	Memory Savings
Attention Slicing	pipe.enable_attention_slicing()	✅ Verified	Medium
VAE Slicing	pipe.enable_vae_slicing()	✅ Verified	Low
VAE Tiling	pipe.enable_vae_tiling()	✅ Verified	High (high resolution)
BF16 inference	torch_dtype=torch.bfloat16	✅ Recommended	About 50%
Sequential CPU Offload	pipe.enable_sequential_model_cpu_offload()	⚠️ Experimental	Very high
Model CPU Offload	pipe.enable_model_cpu_offload()	⚠️ Experimental	High

# 使用内存优化运行
python scripts/run_pipeline.py \
    --model ./fake_flux_dev \
    --prompt "a landscape" \
    --device npu:0 --dtype bfloat16 \
    --steps 20 \
    --attention-slicing --vae-tiling \
    --output optimized.png --benchmark

Detailed guide:references/memory-optimization.md

5. LoRA Integration

Load LoRA adapters to enhance generation results:

import torch
import torch_npu

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("./my_model", torch_dtype=torch.bfloat16)
pipe = pipe.to("npu:0")

# 加载 LoRA
pipe.load_lora_weights("./my_lora", adapter_name="style")

# 生成（控制 LoRA 强度）
image = pipe(
    prompt="a portrait in oil painting style",
    num_inference_steps=20,
    cross_attention_kwargs={"scale": 0.8},
).images[0]

Multiple LoRA stacking:

pipe.load_lora_weights("./lora_style", adapter_name="style")
pipe.load_lora_weights("./lora_detail", adapter_name="detail")
pipe.set_adapters(["style", "detail"], adapter_weights=[0.7, 0.3])

Use script:

python scripts/run_pipeline.py \
    --model ./my_model \
    --prompt "a portrait" \
    --device npu:0 --dtype bfloat16 \
    --lora ./my_lora --lora-scale 0.8 \
    --output lora_output.png

Detailed guide:references/lora-guide.md

6. Multi-Card Parallel Inference (Context Parallel)

Starting from diffusers>=0.36.0 , you can refer to the Parallel API to perform Context Parallel across multiple cards; the core entry point is:

• ContextParallelConfig
• ParallelConfig
• apply_context_parallel

Ascend NPU needs to use the torch.distributed + hccl backend to start multiple processes.

import os

import torch
import torch.distributed as dist
import torch_npu

from diffusers import ContextParallelConfig, DiffusionPipeline


def setup_dist():
    if not dist.is_initialized():
        dist.init_process_group(backend="hccl")
    local_rank = int(os.environ["LOCAL_RANK"])
    world_size = dist.get_world_size()
    torch.npu.set_device(local_rank)
    return local_rank, world_size


local_rank, world_size = setup_dist()
device = f"npu:{local_rank}"

pipe = DiffusionPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
).to(device)

# NPU 推荐使用 _native_npu；CUDA 示例通常使用 _native_cudnn
pipe.transformer.set_attention_backend("_native_npu")

cp_config = ContextParallelConfig(ulysses_degree=world_size)
pipe.transformer.enable_parallelism(config=cp_config)

image = pipe(
    prompt="a tiny astronaut hatching from an egg on the moon",
    guidance_scale=3.5,
    num_inference_steps=30,
).images[0]

if dist.get_rank() == 0:
    image.save("flux_cp_output.png")

dist.destroy_process_group()

In a single-machine 2-card example, you can use:

torchrun --nproc_per_node=2 scripts/run_context_parallel.py \
    --model ./fake_flux_dev \
    --prompt "a tiny astronaut hatching from an egg on the moon" \
    --parallel-mode context \
    --device-type npu --backend hccl \
    --attention-backend _native_npu \
    --steps 20 --output flux_cp_output.png

If the current backend does not yet support context parallel (for example, due to limitations of the underlying attention kernel), you can fall back to multi-process data parallel:

torchrun --nproc_per_node=2 scripts/run_context_parallel.py \
    --model ./fake_flux_dev \
    --prompt "a tiny astronaut hatching from an egg on the moon" \
    --parallel-mode data \
    --device-type npu --backend hccl \
    --steps 20 --output flux_dp_output.png

By default, it is recommended to use Ulysses Attention directly:

cp_config = ContextParallelConfig(ulysses_degree=world_size)
pipe.transformer.enable_parallelism(config=cp_config)

If needed, switch to Ring Attention based on the model/backend characteristics:

cp_config = ContextParallelConfig(ring_degree=world_size)
pipe.transformer.enable_parallelism(config=cp_config)

If further tuning is needed, please first refer to the official attention backend documentation:

• https://huggingface.co/docs/diffusers/v0.36.0/en/optimization/attention_backends
• Mirror:https://hf-mirror.com/docs/diffusers/v0.36.0/en/optimization/attention_backends

Version note:

• diffusers>=0.36.0: you can use api/parallel。
• diffusers<=0.35.2: there is no docs/source/en/api/parallel.mdin the repository path, so you should fall back to regular multi-process parallelism (splitting requests across cards) or upgrade the version.

Detailed process:references/distributed-inference.md

7. API documentation index and version-based retrieval

When a user asks "how do I use a certain API", do not guess the URL first; follow the process below to search:

1. Read the user's actual version (for example, 0.35.2、0.36.0）。
2. Enumerate the API file tree for the corresponding Tag:docs/source/en/api/**。
3. Locate the specific path based on the user's question (for example, parallel.md、pipelines/flux.md）。
4. Then map it to the official documentation URL; if the official website is unreachable, fall back to hf-mirror。

Path mapping rules:

• GitHub source file:
  • https://github.com/huggingface/diffusers/blob/v{version}/docs/source/en/api/{path}.md
• Official documentation:
  • https://huggingface.co/docs/diffusers/v{version}/en/api/{path}
• Mirror documentation:
  • https://hf-mirror.com/docs/diffusers/v{version}/en/api/{path}

Example:parallel.md exists in v0.36.0 , but does not exist in the API path of v0.35.2 .

Detailed index:references/api-navigation.md

Performance Benchmark Testing

Use the benchmark script to measure inference performance:

python scripts/benchmark_pipeline.py \
    --model ./model_weights \
    --prompt "a photo of a cat" \
    --num-runs 5 \
    --warmup-runs 1 \
    --output-json benchmark_results.json

Key metrics:

Metric	Description
First inference latency	Includes graph compilation/cache construction, usually slower
Average inference latency	Steady-state performance (excluding warm-up)
P50 / P95 latency	Latency distribution
NPU memory peak	torch.npu.max_memory_allocated()
Throughput	images/sec or frames/sec

Script reference

run_pipeline.py

General Pipeline inference script, supports any Diffusers model:

Parameter	Required	Default	Description
--model	Yes	-	Model path (local directory or HF model ID)
--prompt	Yes	-	Generation prompt
--device	No	npu:0	Target device
--dtype	No	bfloat16	Data type: float32, float16, bfloat16
--steps	No	20	Number of inference steps
--seed	No	42	Random seed
--output	No	output.png	Output file (.png image / .mp4 video)
--width	No	Model default	Output width
--height	No	Model default	Output height
--guidance-scale	No	3.5	Classifier-free guidance
--lora	No	-	LoRA weights path
--lora-scale	No	1.0	LoRA strength
--attention-slicing	No	Off	Enable attention slicing
--vae-slicing	No	Off	Enable VAE slicing
--vae-tiling	No	Off	Enable VAE tiling
--cpu-offload	No	Off	Enable sequential CPU offload
--benchmark	No	Off	Print detailed timing metrics

validate_pipeline.py

Pre-inference precheck script:

Parameter	Required	Default	Description
--model	No	-	Model path (checks weight structure when provided)
--device	No	npu:0	Target device
--min-memory	No	16	Minimum free NPU memory (GB)

benchmark_pipeline.py

Detailed performance benchmark script:

Parameter	Required	Default	Description
--model	Yes	-	Model weight path
--prompt	Yes	-	Generation prompt
--device	No	npu:0	Device
--dtype	No	bfloat16	Data type
--steps	No	20	Inference steps
--warmup-runs	No	1	Warmup count
--num-runs	No	5	Test count
--attention-slicing	No	Off	Enable attention slicing
--vae-slicing	No	Off	Enable VAE slicing
--vae-tiling	No	Off	Enable VAE tiling
--output-json	No	-	Result JSON output path

run_context_parallel.py

Multi-card context parallel inference script (requires torchrun）：

Parameter	Required	Default	Description
--model	Yes	-	Local model path
--prompt	Yes	-	Generation prompt
--parallel-mode	No	context	context(Parallel API) or data(multiprocess parallel)
--device-type	No	npu	npu/cuda/cpu
--backend	No	hccl	Distributed backend
--attention-backend	No	Auto-selected by device	NPU default _native_npu, CUDA default _native_cudnn
--ring-degree	No	-	Ring Attention parallelism
--ulysses-degree	No	Auto (default=world_size)	Ulysses Attention parallelism
--dtype	No	bfloat16	Data type
--steps	No	20	Inference steps
--output	No	cp_output.png	Rank0 output image

Common model reference

Model	Pipeline type	Recommended dtype	Recommended NPU memory
FLUX.1-dev	FluxPipeline	bfloat16	≥24 GB
SDXL	StableDiffusionXLPipeline	float16	≥8 GB
SD 3.5	StableDiffusion3Pipeline	bfloat16	≥16 GB
Wan 2.1	WanPipeline	bfloat16	≥24 GB
CogVideoX	CogVideoXPipeline	bfloat16	≥24 GB

FAQ

Insufficient NPU memory (OOM)

RuntimeError: NPU out of memory

Solution: Enable memory optimization (--attention-slicing --vae-tiling), use bfloat16, reduce resolution or steps.

First inference is very slow

The first NPU inference requires compiling the computation graph; subsequent inference speeds are normal. The script's --benchmark mode automatically performs warmup.

Generator device error

RuntimeError: Expected a 'npu' device type for generator

Solution: Use torch.Generator("npu") instead of torch.Generator("cpu")。

CPU Offload does not take effect

accelerate device hooks may not fully support npu. It is recommended to prioritize attention slicing and VAE tiling.

More issues:references/troubleshooting.md

Reference resources

• Environment setup - Phase #1: CANN + torch_npu environment setup
• Weight preparation - Phase #2: Model download and fake weight generation
• Memory optimization details - Complete guide to NPU memory optimization
• LoRA details - LoRA loading, multiple LoRA, weight fusion
• Multi-card parallel inference - Context Parallel and torchrun multi-card execution
• API search index - Locate Diffusers API paths and documentation URLs by version
• Troubleshooting - Common issues and fixes for NPU inference
• Diffusers official documentation
• Ascend PyTorch extension