SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

LLaDA 2.0 - Ant Group's Open-Source Discrete Diffusion Large Language Model

Author: AI小集 · Original source: AI工具集

What is LLaDA 2.0

LLaDA 2.0 is a discrete diffusion large language model (dLLM) open-sourced by Ant Group, comprising two versions: 16B (mini) and 100B (flash). It breaks the bottleneck that diffusion models are difficult to scale, and for the first time scales the parameter size to the 100B level. Through the innovative Warmup-Stable-Decay (WSD) continual pre-training strategy, LLaDA 2.0 achieves a smooth transition from autoregressive (AR) models to diffusion models, inherits the knowledge of AR models, and avoids the high cost of training from scratch.

LLaDA 2.0

Main Features of LLaDA 2.0

  • Large-scale Parameter Scaling: LLaDA 2.0 offers 16B and 100B versions, making it the largest diffusion language model to date, breaking through the limitation that diffusion models are difficult to scale up.
  • Efficient Inference Acceleration: With the parallel decoding mechanism, LLaDA 2.0's inference speed reaches up to 535 tokens/s, 2.1 times faster than autoregressive models of the same level, significantly improving generation efficiency.
  • Smooth Transition and Knowledge Inheritance: Adopting the Warmup-Stable-Decay (WSD) strategy, it achieves a smooth transition from autoregressive models to diffusion models, inherits the knowledge of AR models, and avoids the cost of training from scratch.
  • Outstanding Performance: It shows significant advantages in structured generation tasks such as code generation, mathematical reasoning, and agent tasks, while matching open-source AR models in other domains.
  • Fully Open Source and Shared: LLaDA 2.0's model weights (16B/100B) and related training code have been fully open-sourced on Hugging Face, making it convenient for developers to use and further research.

Technical Principles of LLaDA 2.0

  • Diffusion Model Architecture: LLaDA 2.0 is based on the Diffusion Model, generating text through gradual denoising. Unlike traditional autoregressive generation, it can decode multiple tokens in parallel, thereby improving generation speed.
  • Mixture of Experts Architecture (MoE): Combined with the MoE architecture, only a portion of parameters (about 1.44 billion) are activated in each inference, significantly reducing computational cost while maintaining high performance.
  • Warmup-Stable-Decay (WSD) Strategy: Through three-stage pre-training (gradually increasing block size, full-sequence training, reducing block size), it achieves a smooth transition from autoregressive models to diffusion models, inherits existing knowledge, and optimizes inference efficiency.
  • Confidence-Aware Parallel Training (CAP): During parallel decoding, an auxiliary loss function rewards tokens that are 'predicted correctly and with high confidence', improving decoding efficiency and achieving high-speed inference.
  • Diffusion Model Version of DPO: Using the Evidence Lower Bound (ELBO) to approximate conditional probabilities, it adapts preference learning (DPO) to diffusion models, optimizing model outputs to align with human preferences.
  • Document-Level Attention Mask: When training with concatenated multiple documents, a document-level attention mask is designed to avoid erroneous connections between unrelated documents, ensuring semantic coherence in long texts.

Project Address of LLaDA 2.0

  • Hugging Face Model Repository：https://huggingface.co/collections/inclusionAI/llada-20
  • Technical Report:https://github.com/inclusionAI/LLaDA2.0/blob/main/tech_report.pdf

Application Scenarios of LLaDA 2.0

  • Code Generation: LLaDA 2.0 performs excellently in code generation tasks, capable of generating high-quality code snippets and helping developers quickly implement functions.
  • Mathematical Reasoning: The model demonstrates strong capabilities in mathematical problem solving and complex reasoning tasks, and can be used in fields such as education and scientific research.
  • Agent tasks: Supports complex agent calls and long-text tasks, suitable for scenarios requiring multi-step reasoning and tool invocation.
  • Text generation: Can generate high-quality text content, suitable for scenarios such as creative writing and content generation.
  • Knowledge question answering: Performs well in knowledge understanding and question-answering tasks, and can be used in fields such as intelligent customer service and knowledge graphs.