FlashQLA - Tongyi Laboratory's Open-Source High-Performance Linear Attention Operator Library

What Is FlashQLA

FlashQLA is an open-source high-performance linear attention operator library implemented by Tongyi Laboratory based on TileLang. Through operator fusion, Gate-driven intra-card sequence parallelism, and Warp-Specialized optimization, FlashQLA achieves 2–3× forward and 2× backward acceleration on Hopper compared with FLA Triton, covering models from 2B to 397B, and improving pretraining and on-device inference efficiency. FlashQLA requires an SM90, CUDA 12.8+, and PyTorch 2.8+ environment.

FlashQLA's Main Features

• High-Performance Linear Attention Operator Library: Deeply optimized for the Gated Delta Network (GDN) attention layers across the entire Qwen series.
• Operator Fusion Acceleration: Performs reasonable operator fusion and performance optimization on the forward and backward processes of GDN Chunked Prefill.
• Full-Spec Model Coverage: Supports multi-spec models from 2B to 397B, covering TP1 to TP8 scenarios.
• Dual-Level API Interface: Provides a high-level API aligned with the FLA signature, as well as low-level fwd / bwd entry points.
• Variable-Length Sequence Support: Built-in varlen variable-length sequence processing capability, adapting to real training and inference data distributions.

FlashQLA's Technical Principles

• TileLang Warp-Specialized Kernel: Builds key fused kernels based on TileLang, and adopts warpgroup specialization to overlap data movement, Tensor Core computation, and CUDA Core computation.
• Automated Intra-Card Sequence Parallelism (AutoCP): Leverages the exponential decay property of the GDN gate to automatically enable intra-card sequence parallelism in scenarios such as TP, long sequences, and small head counts, improving GPU SM utilization.
• Sliding Window Warmup Mechanism: For linear attention heads with decay properties, only 6–8 chunks of warmup are needed to accurately obtain the subsequence initial state, discarding the computation of the correction term M matrix.
• Hardware-Friendly Algebraic Rewriting: Performs algebraic transformations on the forward and backward processes of GDN Chunked Prefill, effectively reducing Tensor Core, CUDA Core, and SFU overhead without affecting numerical precision.
• A Compromise Architecture Balancing Memory Access and Parallelism: Splits the computation process into two fused kernels and inserts CP preprocessing in between, avoiding the problem of low GPU utilization of a fully-fused kernel in small batch / TP scenarios.

How to Use FlashQLA

• Environment Check: Confirm that the hardware is NVIDIA SM90 (Hopper architecture) and that the software environment meets the requirements of CUDA 12.8+ and PyTorch 2.8+.
• Installation and Deployment: Clone the FlashQLA repository from GitHub and complete compilation and installation via pip.
• Module Import: Import in Python chunk_gated_delta_rule function.
• Data Preparation: Prepare the input tensors q, k, v as well as the gate parameters g and beta, ensuring that the shape of each tensor meets the interface requirements.
• Execute Computation: Call chunk_gated_delta_rule and pass in the corresponding parameters to obtain the output result O and the final state.
• Advanced Configuration: If variable-length sequences need to be processed, pass in the cu_seqlens parameter; if state continuation is needed, pass in initial_state。
• Automatic Optimization: AutoCP sequence parallelism is automatically triggered according to batch size and sequence length, and no manual configuration is required.

Key information and usage requirements of FlashQLA

• Publisher: Tongyi Lab / QwenTeam
• Open-source address：github.com/QwenLM/FlashQLA
• Hardware requirements: NVIDIA SM90 (Hopper architecture, such as H200)
• Software requirements：CUDA 12.8+，PyTorch 2.8+
• Supported models: Qwen3.5 / Qwen3.6 series (head dim covers 64 to 8, corresponding to TP1 to TP8)
• Acceleration effect: Forward 2–3×, backward 2× (compared with FLA Triton Kernel)

Core advantages of FlashQLA

• A compromise architecture balancing memory access and parallelism: The computation is split into two fused kernels with CP preprocessing inserted in between, avoiding the low GPU utilization problem of a fully-fused kernel in small-batch / TP scenarios, and reducing the memory access overhead of repeatedly reading and writing intermediate variables in HBM through reasonable splitting.
• AutoCP automatic enabling mechanism: Only at batch_size × num_heads ≤ 40 or batch_size × num_heads ≤ 56 且 seq_len ≥ 8192 , intra-card sequence parallelism is automatically triggered, avoiding unnecessary redundant computation and adaptively balancing parallelism and memory access cost.
• Sliding window warmup mechanism: Using the exponential decay property of the GDN gate, only 6–8 chunks of warmup are needed for 60–80% of the linear attention heads to accurately obtain the subsequence initial state, directly discarding the computation of the correction term M matrix, greatly reducing the CP preprocessing overhead.
• Warp-Specialized computation overlap: Based on TileLang's warpgroup specialization design, producer and consumer warpgroups cooperate within the same SM, covering data movement and Tensor Core / CUDA Core computation through a ping-pong structure.
• Hardware-friendly algebraic rewriting: Algebraic transformation and simplification are performed on the forward and backward processes, effectively reducing the hardware overhead of Tensor Core, CUDA Core, and SFU without affecting numerical precision.

Project addresses of FlashQLA

• Project official website：https://qwen.ai/blog?id=flashqla
• GitHub repository：https://github.com/QwenLM/FlashQLA

Comparison of FlashQLA with similar competing products

Application Scenarios of FlashQLA

• Ultra-large model pretraining: Covers the full Qwen model series including 397B / 122B / 35B / 27B, supports 256K long-context training, and significantly reduces the compute and time overhead of attention layers in end-to-end training
• On-device agentic inference: Targets chunked prefill scenarios with batch_size=1 and small models (such as 2B / 0.8B), improving GPU utilization with small head counts through AutoCP and accelerating real-time responses of on-device Agents
• Large model online deployment: Handles long-sequence inputs such as coding agents in TP (Tensor Parallelism) scenarios, addressing the GPU utilization bottleneck when chunked prefill cannot achieve a sufficiently large batch, and improving service throughput
• General GDN / linear attention architecture acceleration: Applies to LLM training and inference based on any Gated Delta Network or linear attention architecture, providing an out-of-the-box high-performance operator replacement solution