dllm

dLLM is a library for unified diffusion language model training and evaluation, dedicated to bringing transparency and reproducibility to the entire development workflow.

Purpose: dLLM is aimed at research and development of diffusion language models, providing a complete toolchain from training and inference to evaluation, helping users reproduce and extend diffusion language models in a standardized way.

Main capabilities:

• Scalable training pipeline: based on the transformers Trainer, supports distributed training methods such as LoRA, DeepSpeed, and FSDP, and supports data subsets, dataset concatenation, 4-bit quantization, streaming loading, and more.
• Unified evaluation pipeline: based on lm-evaluation-harness, abstracts away inference details and simplifies custom evaluation.
• Provides minimal training/inference/evaluation recipes for open-weight models (such as LLaDA and Dream).
• Implements multiple training algorithms: MDLM (masked diffusion), BD3LM (block diffusion), Edit Flows, and more.
• Supports diffu-GRPO training (for masked diffusion language models, validated on LLaDA and Tiny-A2D, covering five reasoning tasks: GSM8K, MATH, Countdown, Sudoku, and Code).
• Supports Fast-dLLM accelerated inference and evaluation (caching, confidence-threshold decoding, etc.).
• Provides the Tiny-A2D small models (0.5B/0.6B) and open recipes for converting any autoregressive model (such as Qwen, LLaMA, and GPT-2) into a diffusion model.
• Provides BERT-Chat and open recipes for converting any BERT encoder (such as BERT, RoBERTa, and ModernBERT) into a diffusion model.

Usage:

• Training: can be launched locally with accelerate, or submitted to a Slurm cluster via sbatch. Typical training entry scripts include examples/llada/sft.py. Supports controlling datasets, quantization, LoRA, distributed configuration, and more through command-line arguments.
• Inference: provides a unified sampler; typical inference entry scripts include examples/llada/sample.py, and also provides interactive chat scripts (such as examples/llada/chat.py).
• Evaluation: can be run based on lm-evaluation-harness, provides automatic evaluation scripts (such as examples/llada/eval.sh), and also supports Fast-dLLM evaluation scripts.
• Example directories: examples/llada (LLaDA/LLaDA-MoE pretraining, fine-tuning, evaluation), examples/llada2 (LLaDA2.0 inference), examples/llada21 (LLaDA2.1 inference), examples/dream (Dream pretraining, fine-tuning, evaluation), examples/a2d (fine-tuning autoregressive models into masked/block diffusion), examples/bert (fine-tuning BERT into a lightweight chatbot), examples/editflow (Edit Flows training reference), examples/fastdllm (Fast-dLLM inference and evaluation), examples/rl (GRPO training).

Applicable scenarios: research, training, fine-tuning, inference acceleration, and standardized evaluation of diffusion language models, especially suitable for developers and researchers who need to reproduce or extend models such as LLaDA and Dream.

Limitations: the materials do not provide specific hardware requirements, model scale limits, license information, or pricing models; this information is not currently available.