What is Dream-7B
Dream-7B is a diffusion reasoning model jointly launched by the University of Hong Kong and Huawei Noah's Ark Lab, and is currently the most powerful open-source diffusion large language model. Dream-7B's training data covers text, mathematics, and code, with pretraining using 580 billion tokens and taking 256 hours. Dream-7B performs excellently on general, mathematical, and programming tasks, comparable to autoregressive models of the same size (Qwen2.5 7B, LLaMA3 8B), and in some cases outperforms the latest Deepseek V3 671B. The model uses a masked diffusion paradigm, based on bidirectional context modeling and flexible controllable generation capabilities, significantly improving the global coherence of generated text.
Dream-7B's Main Features
- Powerful text generation capability: Performs excellently on general text, mathematics, and programming tasks, surpassing autoregressive models of the same size.
- Flexible generation method: Supports text generation in any order, allowing users to specify the generation order according to their needs.
- Efficient planning capability: Performs outstandingly in tasks requiring multi-step planning, such as Countdown and Sudoku tasks.
- Adjustable generation quality: Supports users in adjusting the number of diffusion steps to balance generation speed and quality.
Dream-7B's Technical Principles
- Diffusion model architecture: Based on Discrete Diffusion Models (DMs), unlike traditional autoregressive models, diffusion models start from a fully noised state and gradually denoise to generate text. The diffusion model architecture supports bidirectional context modeling, integrating forward and backward information, significantly improving the global coherence of generated text.
- Masked diffusion paradigm: The model uses a masked diffusion paradigm, predicting all masked tokens to gradually denoise. This supports the model in better aligning with the weights of autoregressive models during training, accelerating the training process.
- Autoregressive model initialization: Based on using the weights of autoregressive models (such as Qwen2.5) as initialization, which is more effective than training diffusion models from scratch and accelerates the training of diffusion models.
- Context-adaptive noise rescheduling: Introduces a context-adaptive noise rescheduling mechanism that dynamically adjusts the noise level based on the context information of each token. The model controls the learning process of each token more finely, improving training efficiency.
- Flexible decoding strategy: During the inference stage, the diffusion model flexibly adjusts the generation order and number of diffusion steps, achieving a dynamic balance between speed and quality.
Dream-7B's Project Address
- Project official website：https://hkunlp.github.io/blog/2025/dream/
- GitHub repository：https://github.com/HKUNLP/Dream
- HuggingFace model library：https://huggingface.co/Dream-org
- Online experience Demo：https://huggingface.co/spaces/multimodalart/Dream
Dream-7B's Application Scenarios
- Text generation and creation: Creates high-quality general text, such as news reports, story creation, copywriting, etc., providing rich and coherent text content.
- Mathematical problem solving: Efficiently solves complex mathematical problems, including the derivation of mathematical problems, formula generation, etc., providing auxiliary tools for education and scientific research.
- Programming assistance: Generates programming code, helping developers quickly build code frameworks, solve programming difficulties, and improve programming efficiency.
- Complex task planning: Used in scenarios requiring multiple constraints and multi-step reasoning, such as task scheduling, path planning, etc.
- Flexible text processing: Adjust generation speed and quality according to needs, suitable for various applications requiring flexible text processing.