SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

LLaDA2.2-flash - Diffusion Language Model Open-Sourced by InclusionAI

Author: AI小集 · Original source: AI工具集

What is LLaDA2.2-flash

LLaDA2.2-flash is a diffusion language model (dLLM) open-sourced by InclusionAI. It adopts a MoE architecture with a total parameter count of approximately 100B and native support for a 128K context. The model fits answers through multiple rounds of parallel denoising and introduces Levenshtein editing to enable insertion, deletion, and modification capabilities for sequences. The model achieves an average score of 53.83 on 7 Agent benchmarks, approaching the 55.74 score of the same-scale autoregressive model Ling-2.6-flash, while maintaining a 1.64x decoding throughput advantage.

LLaDA2.2-flash

Main Features of LLaDA2.2-flash

  • Diffusion-based Parallel Generation: Processes multiple Token positions simultaneously through block parallel decoding, achieving high-throughput text generation.
  • Levenshtein Editing: Supports four editing operations during denoising—KEEP, SUBSTITUTE, DELETE, and INSERT—breaking through the fixed-length replacement limitation.
  • Long Context Processing: Natively supports a 128K context window, adapting to complex scenarios such as long-range software engineering and tool-calling trajectories.
  • Agent Task Execution: Supports environment-feedback tasks requiring continuous error correction, such as code repair, API interaction, and multi-turn tool calling.
  • MoE Efficient Inference: Adopts a mixture-of-experts architecture and controls inference cost under long contexts through a block routing mechanism.

Technical Principles of LLaDA2.2-flash

  • Diffusion Language Model Paradigm: Unlike autoregressive models that generate Token by Token from left to right, LLaDA2.2 first determines a batch of noise positions to be filled, then gradually fits a complete sequence through multiple rounds of denoising. Multiple positions advance collaboratively, offering the potential for repeated refinement.
  • Levenshtein Editing Mechanism: Uses the longest common subsequence (LCS) to align generated drafts with target sequences, constructing four types of editing instructions: KEEP, SUBSTITUTE, DELETE, and INSERT. DELETE removes redundant Tokens and shifts subsequent sequences left, while INSERT creates a [MASK] before the current position awaiting subsequent filling, enabling the model to correct sequences of non-equal length.
  • L-EBPO Reinforcement Learning: Proposes a block-level policy optimization algorithm based on the Levenshtein editing evidence lower bound. The outer layer optimizes trajectory-level decisions for multi-turn Agent interactions, while the inner layer handles insertions and deletions within Blocks during a single generation. By recovering editing trajectories, gradients cover structural decisions, and environmental rewards come from tool-calling correctness, output format validity, and task completion.
  • Block Routing Mechanism: To address the expert union expansion problem of MoE in long-context block diffusion, Token racing is used to obtain Block-level admission scores. From 256 routed experts, Top-48 are selected to form a candidate pool, and then Top-k routing is performed within the pool for each Token, giving the expert working set a predictable O(C) upper bound.
Dig into GitHub

Follow WeChat and reply with "Open Source" to join theAI Open Source Project Exchange Group

How to Use LLaDA2.2-flash

  • Hardware Preparation: Local deployment of LLaDA2.2-flash requires at least 4 A100-80GB GPUs or equivalent VRAM GPUs to support BF16 full-precision loading. If using FP8 quantization or handling 128K long-context scenarios, correspondingly more VRAM resources need to be reserved.
  • Model Download: When downloading the model via the ModelScope SDK, executefrom modelscope import snapshot_download; model_dir = snapshot_download("inclusionAI/LLaDA2.2-flash")to automatically complete the download of approximately 206GB of model files.
  • Git Clone: If obtaining the model via Git LFS, rungit lfs installto initialize large file support, then executegit clone https://modelscope.cn/models/inclusionAI/LLaDA2.2-flash.gitPull the complete model repository.
  • Model loading: When loading the model with Transformers, callAutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, device_map="auto")and be sure to settrust_remote_code=Trueto enable the custom diffusion decoding logic.
  • Precision setting: After the model is loaded, executemodel = model.to(torch.bfloat16)to convert it to BF16 precision, then callmodel.eval()to switch the model to evaluation inference mode.
  • Tokenizer loading: Load the corresponding tokenizerAutoTokenizer.from_pretrained(model_path, trust_remote_code=True)at the same time to ensure the tokenization rules fully match the model configuration.
  • Input construction: After constructing the user prompt, usetokenizer.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True, tokenize=True, return_tensors="pt")to format the conversation into the input tensors required by the model.
  • Parameter configuration: When callingmodel.generate(), configure the key diffusion decoding parameters:block_length=32controls the number of parallel denoising tokens per step,threshold=0.5sets the denoising confidence threshold,editing_threshold=0.0disables active editing,temperature=0.0enables greedy decoding,gen_length=512sets the maximum generation length.
  • Result decoding: After generation is complete, usetokenizer.decode(generated_tokens[0], skip_special_tokens=True)to decode the token sequence into readable text and output the final result.

Core advantages of LLaDA2.2-flash

  • High-throughput decoding: The average decoding throughput in BF16 reaches 1.64 times that of the same-scale autoregressive model Ling-2.6-flash, and FP8 quantization can further improve it by 18.6%.
  • Structured self-correction: Levenshtein editing gives the model the ability to add, delete, and modify, raising the SWE-bench Verified resolution rate from 35.8 to 44.4, an absolute gain of 8.6 percentage points.
  • Native long context: Extended from 8K to 128K through continued pretraining, a non-extrapolation approach, adapting to the long-range interaction needs of Agents.
  • Open source and reproducible: Licensed under Apache-2.0, with both the model and training details made public, supporting complete local reproduction.
  • MoE inference efficiency: The block routing mechanism keeps the expert working set of each Block within a fixed capacity, significantly reducing HBM traffic and communication costs.

Project addresses of LLaDA2.2-flash

  • GitHub repository：https://github.com/inclusionAI/LLaDA2.X
  • ModelScope address：https://modelscope.cn/models/inclusionAI/LLaDA2.2-flash

Comparison of LLaDA2.2-flash with similar competitors

Comparison dimension LLaDA2.2-flash Ling-2.6-flash
Architectural paradigm Adopts a diffusion language model (dLLM) combined with a MoE architecture, generating sequences through multiple rounds of parallel denoising. Adopts autoregression (AR) combined with a MoE architecture, generating text token by token from left to right.
Sequence editing capability Natively supports Levenshtein editing, with four add, delete, and modify operations: KEEP, SUBSTITUTE, DELETE, and INSERT. No native sequence editing capability; post-generation correction depends on external resampling or post-processing.
Reinforcement learning strategy Uses the L-EBPO algorithm to jointly optimize trajectory-level decisions and intra-block editing actions in environmental feedback. Adopts standard RLHF or a traditional reinforcement learning framework for alignment and optimization.
Agent benchmark average score Achieves an average score of 53.83 across 7 Agent benchmark tests. Achieves an average score of 55.74 across the same 7 Agent benchmark tests, 1.91 points higher.
Decoding throughput BF16 average decoding throughput reaches 1.64 times that of Ling-2.6-flash, and increases by another 18.6% after FP8 quantization. As an autoregressive baseline model, decoding throughput is limited by the token-by-token generation method.
Context window Native support for a 128K context through continual pretraining, not positional extrapolation. The context capability's specific training method is not disclosed in detail in public technical reports.

Application scenarios of LLaDA2.2-flash

  • Software engineering Agent: The model natively supports Levenshtein-style insertion, deletion, and modification capabilities, and is competent for complex programming tasks requiring non-equal-length text editing, such as code repair, bug localization, and function completion.
  • Multi-turn tool calling: In API interactions and MCP protocol execution, the model can dynamically correct parameters and add or remove fields, adapting to complex Agent workflows that require continuous error correction.
  • Long document processing: With a native 128K context window, the model can efficiently perform long-text analysis, report generation, and knowledge extraction without relying on positional extrapolation.
  • Scientific research and experiment reproduction: As an open-source MoE diffusion language model, LLaDA2.2-flash provides academia with a reproducible research foundation and an alternative technical path beyond the autoregressive route.