What is LLaDA
LLaDA (Large Language Diffusion with mAsking) is a new type of large language model jointly launched by the team of Professors Li Chongxuan and Wen Jirong from the Gaoling School of Artificial Intelligence at Renmin University of China and Ant Group. It is based on a diffusion model framework rather than the traditional autoregressive model (ARM). LLaDA models text distribution based on a forward masking process and a reverse recovery process, using a Transformer as the masking predictor, and optimizes the likelihood lower bound to achieve generation tasks. In the pretraining stage, LLaDA uses 2.3 trillion tokens of data, and improves instruction-following ability based on supervised fine-tuning (SFT). LLaDA performs excellently in scalability, in-context learning, and instruction following, and solves the "reversal curse" problem of traditional ARMs in reversal reasoning tasks. Its 8B parameter version is comparable to strong models such as LLaMA3 in multiple benchmark tests, demonstrating the great potential of diffusion models as an alternative to autoregressive models.
Main Features of LLaDA
- Efficient Text Generation: It can generate high-quality, coherent text content, suitable for scenarios such as writing, dialogue, and content creation.
- Powerful In-Context Learning Ability: It quickly adapts to new tasks based on context.
- Instruction-Following Ability: It can better understand and execute human instructions, suitable for scenarios such as multi-turn dialogue, question answering, and task execution.
- Bidirectional Reasoning Ability: It solves the "reversal curse" of traditional autoregressive models and performs excellently in both forward and reverse reasoning tasks, such as poetry completion tasks.
- Multi-Domain Adaptability: It performs excellently in multiple fields such as language understanding, mathematics, programming, and Chinese understanding, and has broad applicability.
Technical Principles of LLaDA
- Diffusion Model Framework: It models text distribution based on a forward masking process (gradually masking tokens in the text) and a reverse recovery process (gradually recovering the masked tokens). It supports the model in generating text in a non-autoregressive way, avoiding the sequential generation limitation of traditional autoregressive models.
- Masking Predictor: It uses an ordinary Transformer architecture as the masking predictor, takes a partially masked text sequence as input, and predicts all masked tokens. This allows the model to capture bidirectional dependencies, rather than only one-way left-to-right generation.
- Optimizing the Likelihood Lower Bound: The model is trained based on optimizing the likelihood lower bound, which is principled in generative modeling and ensures the model's scalability and generation ability under large-scale data and model parameters.
- Pretraining and Supervised Fine-Tuning: LLaDA is based on a combination of pretraining and supervised fine-tuning (SFT). In the pretraining stage, large-scale text data is used for unsupervised learning, while in the SFT stage, annotated data is used to improve the model's instruction-following ability.
- Flexible Sampling Strategies: During the generation process, LLaDA supports multiple sampling strategies (such as random masking, low-confidence masking, semi-autoregressive masking, etc.), balancing generation quality and efficiency.
LLaDA Project Address
- Project Official Website：https://ml-gsai.github.io/LLaDA
- GitHub Repository：https://github.com/ML-GSAI/LLaDA
- arXiv Technical Paper：https://arxiv.org/pdf/2502.09992
Application Scenarios of LLaDA
- Multi-Turn Dialogue: Used for intelligent customer service, chatbots, etc., supporting smooth multi-turn communication.
- Text Generation: Suitable for writing assistance, creative copywriting, etc., and can generate high-quality text.
- Code Generation: Help developers generate code snippets or fix errors, improving programming efficiency.
- Mathematical Reasoning: Solve mathematical problems and provide solution steps, usable in the education field.
- Language Translation: Enable multilingual translation and promote cross-cultural communication.