Nemotron-Labs-Diffusion - NVIDIA's Open-Source Three-Mode Language Model

What is Nemotron-Labs-Diffusion

Nemotron-Labs-Diffusion is a three-mode language model released by NVIDIA, unifying autoregressive, diffusion, and self-speculative decoding within a single architecture. Trained with a joint AR-diffusion objective, the model can switch modes under different concurrency scenarios to maintain high throughput. The series includes 3B, 8B, and 14B parameter base, instruction, and vision-language models, surpassing existing open-source AR and diffusion language models in both accuracy and speed.

Main Features of Nemotron-Labs-Diffusion

• Three-Mode Decoding Switching: Supports three decoding modes—AR, diffusion, and self-speculative—and can seamlessly switch within a single model by changing the attention pattern.
• Block-Level Diffusion Parallel Generation: Uses a block-level diffusion formulation, splitting the sequence into blocks and performing bidirectional parallel decoding within each block, generating multiple tokens per forward pass.
• Self-Speculative Draft Verification: Diffusion generates multi-token drafts in parallel, and AR verifies them within the same model, sharing the KV cache and requiring no auxiliary prediction head.
• Vision-Language Understanding: Provides vision-language model variants supporting image understanding and image-text reasoning, extending the boundaries of multimodal applications.
• Efficient Inference Deployment: Supports FP8 low-precision formats and, combined with the SGLang framework, achieves high-throughput serving on GB200 GPUs.

Technical Principles of Nemotron-Labs-Diffusion

• Joint AR-Diffusion Objective Training: The model simultaneously optimizes the autoregressive next-token loss and the block-level diffusion denoising loss, balancing the two with a weighted combination (α=0.3). The AR objective provides a left-to-right language prior for diffusion, preventing diffusion training from wasting capacity on arbitrary token permutations; the diffusion objective, in turn, enhances the model's forward-planning ability.
• Two-Stage Training Strategy: In the first stage, only the AR objective is used for pretraining to establish a strong left-to-right language inductive bias; in the second stage, joint training is enabled so that diffusion gradients supplement it, achieving a harmonious fusion of the two objectives.
• Structured Attention Pattern: During training, a dual-stream input is used. In the noisy stream, tokens have bidirectional attention within blocks and causal attention across blocks, and attend to the already-generated prefix in the clean stream; the clean stream maintains a strictly causal mask, allowing the AR objective and the diffusion objective to be computed jointly in the same forward-backward pass.
• Global Loss Averaging: To address the gradient variance problem caused by varying numbers of noisy tokens across samples in the diffusion objective, global token-level loss averaging is used, weighting all tokens in the batch equally to avoid a few high-weight noisy samples disproportionately affecting the batch gradient.
• Optimal Sampling and LoRA Enhancement: The diffusion mode works with a sampler optimized based on sampling trajectories to improve parallelism; the self-speculative mode can be augmented with a LoRA draft adapter to enhance draft quality, increasing the acceptance rate and actual device efficiency.

Follow on WeChat and reply “Open Source” to join theAI Open Source Project Communication Group

How to Use Nemotron-Labs-Diffusion

• Install Dependencies: Run commands such as pip install "transformers>=5.0" torch peft to prepare the Python and GPU environment.
• Obtain the Model: Pull model weights such as nvidia/Nemotron-Labs-Diffusion-8B and the corresponding tokenizer from HuggingFace.
• Select a Mode: Choose AR mode (high concurrency), diffusion mode (maximum parallelism), or self-speculation mode (low latency) based on the concurrency level.
• Run scripts: Use chat_ar.py、chat_dlm.py or chat_linear_spec.py and other official scripts for single-turn or multi-turn dialogue inference.
• Service deployment: Launch the inference service based on the SGLang framework, configure Linear Self-Speculation and FP8 precision, and integrate it into the production environment.

Core advantages of Nemotron-Labs-Diffusion

• Unified three-mode architecture: A single model simultaneously masters AR language priors and diffusion parallel planning capabilities, with no need to maintain multiple models and pipelines.
• Significantly improved throughput: The number of tokens decoded in a single forward pass by the 8B model reaches 6 times that of Qwen3-8B, and throughput on SPEED-Bench on GB200 increases by 4 times.
• Self-speculation outperforms MTP: The acceptance rate and actual device efficiency of self-speculation mode are both better than multi-token prediction (MTP) methods.
• Harmonious dual-objective training: Global loss averaging and a two-stage strategy effectively balance the gradients of the dual objectives and avoid capacity competition between modes.
• Scenario-adaptive switching: By switching the attention mode, it can adapt to high-concurrency cloud-side and low-concurrency device-side scenarios without architectural changes.

Project address of Nemotron-Labs-Diffusion

• HuggingFace model library：https://huggingface.co/collections/nvidia/nemotron-labs-diffusion
• arXiv technical paper：https://arxiv.org/pdf/2607.05722

Comparison of Nemotron-Labs-Diffusion with similar competitors

• Real-time dialogue assistant: In low-concurrency scenarios, use self-speculation mode to achieve low-latency interactive AI dialogue and improve user experience.
• Cloud inference service: AR mode adapts to high-concurrency batch requests, fully utilizes GPU compute density, and reduces cloud deployment costs.
• Code and mathematical reasoning: Diffusion mode enhances forward-looking planning and is suitable for programming assistance and mathematical solving tasks that require multi-step logical reasoning.
• On-device local inference: The 8B model can run on personal devices, and switching among the three modes adapts to different loads while ensuring data privacy.
• Vision-language applications: Vision-language model variants support image understanding and image-text question answering, suitable for intelligent document analysis and multimodal interaction.

Application scenarios of Nemotron-Labs-Diffusion

• Real-time dialogue assistant: In low-concurrency scenarios, use self-speculation mode to achieve low-latency interactive AI dialogue and improve user experience.
• Cloud inference service: AR mode adapts to high-concurrency batch requests, fully utilizes GPU compute density, and reduces cloud deployment costs.
• Code and mathematical reasoning: Diffusion mode enhances forward-looking planning and is suitable for programming assistance and mathematical solving tasks that require multi-step logical reasoning.
• On-device local inference: The 8B model can run on personal devices, and switching among the three modes adapts to different loads while ensuring data privacy.
• Vision-language applications: Vision-language model variants support image understanding and image-text question answering, suitable for intelligent document analysis and multimodal interaction.