What is ELF
ELF (Embedded Language Flows) is the firstdiffusionlanguage model from Kaiming He's team, adopting a continuous diffusion paradigm instead of the traditional autoregressive route. The model denoises and generates text entirely in a continuous embedding space, discretizing into tokens at the final step. With 105M parameters, 45B training tokens, and 32 sampling steps, it achieves a generation perplexity of 24 on OpenWebText, surpassing mainstream discrete diffusion models with nearly 10x less training data, and performs excellently on tasks such as unconditional generation, translation, and summarization.
ELF's main features
-
Continuous-space text generation: Performs denoising entirely in the continuous embedding space, projecting continuous representations into discrete tokens via an unembedding layer at the final time step (t=1).
-
Unconditional language generation: Starting from pure Gaussian noise, generates natural, low-perplexity human-style text through 32 iterations.
-
Conditional text tasks: Supports conditional generation tasks such as machine translation (WMT14) and text summarization (XSum), outperforming existing diffusion language models and some autoregressive baselines.
-
Unified training-inference architecture: The denoising network and the final decoding network share the same set of parameters, switching via a binary mode token (denoising/decoding mode), with no need to separately train an independent decoder.
ELF's technical principles
-
Continuous Embedding encoding: Discrete tokens are first mapped by a T5 pretrained encoder into bidirectional contextual embeddings, used only during training, with no additional modules added at inference time.
-
Flow Matching + x-prediction: Defines a rectified flow trajectory from noise to clean data in continuous space; the network directly predicts the clean embedding (x-prediction), not the velocity field, and the training objective is MSE loss, which is more stable on high-dimensional representations.
-
Final-step Discretization: In the final step, continuous embeddings are projected into token logits through a learnable unembedding matrix; token-level corruption is added during training to prevent the task from being too simple, and the loss is cross-entropy.
-
Self-conditioning CFG: Introduces the idea of classifier-free guidance from image generation, using training-time CFG as the conditioning signal with no inference overhead.
How to use ELF
-
Access the code repository: Visit GitHub to clone the project code.
-
Prepare the environment: Install dependency libraries (PyTorch, etc.) and configure the GPU environment.
-
Data preprocessing: Convert text data into continuous embedding format via the T5 encoder.
-
Model training: Train the denoising network using the Flow Matching + x-prediction objective function, supporting MSE or CE loss.
-
Text generation: Starting from Gaussian noise, call the ELF denoising network for 32 iterations, switching to decoding mode in the final step to output tokens.
-
Downstream task fine-tuning: Perform conditional generation fine-tuning on datasets such as WMT14 and XSum.
ELF's core advantages
-
Extremely high data efficiency: Trained with 45B tokens, an order of magnitude less than the 500B+ tokens of competitors such as MDLM, Duo, and FLM.
-
Very few sampling steps: 32 steps can achieve or even surpass the generation quality of competitors' 1024 steps.
-
Better generation quality: OpenWebText generation perplexity as low as 24, text is more natural with weaker AI traces.
-
Simple and unified architecture: Denoising and decoding share the network, requiring no additional decoder module, avoiding the overhead and error accumulation of decoder training in latent diffusion.
ELF project address
- GitHub repository：https://github.com/lillian039/ELF
- arXiv technical paper：https://arxiv.org/pdf/2605.10938
Comparison of ELF with similar competitors
| Comparison dimension | ELF | MDLM | LLaDA |
|---|---|---|---|
| Technical route | Continuous diffusion (full-process denoising in embedding space, discretization in the final step) | Discrete diffusion (operating directly in token space, maintaining discrete states at each step) | Discrete diffusion (masked diffusion language model based on BERT architecture) |
| Parameter scale | 105M | 350M / 1.3B | 8B |
| Training data | 45B token(one order of magnitude less) | 500B+ token | Trillions of tokens |
| Sampling steps | 32 steps | 1024 steps (standard) / 64 steps (requires distillation) | 64~128 steps |
| Generation perplexity | 24(OpenWebText, 32 steps without distillation) | ~35 (1024 steps) / ~60 (32 steps without distillation) | ~30 (128 steps) |
| Architecture design | Denoising and decoding share the same network, with no additional modules | Discrete state transitions in vocabulary space at each step | Transformer based on masked prediction, predicting masked tokens at each step |
| Core advantages | Extremely high data efficiency, extremely fast sampling, simplest architecture | Naturally fits the discrete nature of language, theoretically intuitive | Large-scale parameters bring strong expressive capability and good scalability |
| Main disadvantages | Depends on a pretrained encoder to provide embeddings | High training data requirements, many sampling steps, generation quality sensitive to the number of steps | High inference cost, requires large amounts of training data, and still requires relatively many inference steps |
Application scenarios of ELF
-
Low-resource efficient text generation: With 45B tokens, a high-quality model can be trained, suitable for enterprises and researchers with limited data budgets to quickly deploy natural language generation capabilities.
-
Machine translation: Surpasses existing diffusion models and some autoregressive baselines on translation tasks such as WMT14, and can serve as the core engine of non-autoregressive translation systems.
-
Text summarization and content rewriting: Performs stably on summarization tasks such as XSum, suitable for scenarios such as news summarization and document distillation that need to preserve key information.
-
Creative writing and dialogue generation: Generation perplexity as low as 24, high naturalness, weak AI traces, suitable for human-like style content creation such as long novels and marketing copy.
-
Research baseline for diffusion language models: First validates the feasibility of the "continuous all the way" route in text generation, providing an important reference and foundational framework for subsequent exploration of large model architectures.