Nemotron-Labs-TwoTower - NVIDIA's Open-Source Two-Tower Architecture Diffusion Language Model

What is Nemotron-Labs-TwoTower

NVIDIA Nemotron-Labs-TwoTower is NVIDIA's open-source two-tower architecture diffusion language model, with approximately 60B total parameters and 3B active parameters. The model decouples context understanding and denoising generation into two independent towers: a frozen AR context tower processes clean tokens, while a trainable diffusion denoising tower refines noisy blocks through cross-attention. The model is built on Nemotron-3-Nano-30B-A3B, trained on approximately 2.1T tokens, retains 98.7% of the quality of the autoregressive baseline, increases generation throughput by 2.42x, and supports three inference modes: Mask Diffusion, Mock-AR, and AR-only.

Main Features of Nemotron-Labs-TwoTower

Two-Tower Decoupled Inference: Decomposes context encoding and denoising generation into two independent modules, avoiding the performance bottleneck of a single network "wearing two hats".
Three Inference Mode Switching: A single checkpoint supports Mask Diffusion, Mock-AR, and AR-only, flexibly adapting to different scenario requirements.
High-Quality Text Generation: Achieves parallelizable iterative generation while retaining 98.7% of the aggregated benchmark quality of the autoregressive baseline.
High-Throughput Generation Acceleration: Generation throughput is increased by 2.42x in a 2×H100 environment, significantly reducing inference latency and computational cost.
Commercial-Grade Open-Source Deployment: Open-sources weights under the NVIDIA Nemotron Open Model License, supporting enterprise commercial use and secondary development.

Technical Principles of Nemotron-Labs-TwoTower

Core Idea: Traditional diffusion language models use a single network to simultaneously take on the two roles of "context understanding" and "iterative denoising," causing the two to constrain each other. TwoTower decouples these two responsibilities into a two-tower architecture:
- Context Tower: A frozen Nemotron-3-Nano-30B-A3B that uses causal attention to process clean tokens, responsible for high-quality context representation encoding.
- Denoiser Tower: A trainable module that uses bidirectional block attention to process noisy token blocks, obtains semantic guidance from the context tower through cross-attention, and progressively refines denoising.
Training Method: Based on a 30B hybrid Mamba-Transformer MoE backbone, trained on approximately 2.1T tokens, using MoE sparse activation to achieve efficient computation.
Inference Advantages: The parallel iterative nature of diffusion models naturally supports accelerated decoding. After the two towers are separated, the denoiser tower does not need to repeatedly encode context, greatly improving wall-clock throughput while ensuring quality.

Follow WeChat and reply "Open Source" to join theAI Open-Source Project Communication Group

How to Use Nemotron-Labs-TwoTower

Visit the HuggingFace Model Page: Open the official repository page for Nemotron-Labs-TwoTower-30B-A3B-Base-BF16.
Read the Model Card and License: Confirm the commercial terms of the NVIDIA Nemotron Open Model License, and understand the model architecture and hardware requirements.
Clone/Download Weights: Use git lfs or the HuggingFace transformers library to download the model weights and configuration files.
Configure the Runtime Environment: Prepare an environment with at least 2×H100 GPUs, and install PyTorch and dependency libraries (refer to the repository requirements.txt）。
Load model and tokenizer: via AutoModelForCausalLM and AutoTokenizer Load the model, and select Mask Diffusion / Mock-AR / AR-only inference mode.
Run inference generation: input a prompt, call the model generation interface, and use the two-tower architecture to achieve efficient parallel denoising generation.
Fine-tuning adaptation: further fine-tune the trainable denoising tower based on your own data, while keeping the context tower frozen and unchanged.

Core advantages of Nemotron-Labs-TwoTower

Two-tower decoupling, each performs its own role: separate context encoding and denoising generation into two independent towers, avoiding the performance bottleneck caused by role conflicts within a single network.
Almost no quality loss: retains 98.7% of the aggregate benchmark quality of the autoregressive baseline, so diffusion generation no longer comes at the cost of sacrificing output quality.
Doubled inference speed: achieves 2.42x wall-clock generation throughput in a 2xH100 environment, significantly reducing latency and compute cost.
One model, three modes: a single checkpoint supports three inference methods—Mask Diffusion, Mock-AR, and AR-only—flexibly adapting to scenarios with different latency and quality requirements.
Open source and commercially usable: releases weights under the NVIDIA Nemotron Open Model License, supporting free enterprise deployment and commercial secondary development.

Project address of Nemotron-Labs-TwoTower

HuggingFace model repository：https://huggingface.co/collections/nvidia/nemotron-labs-twotower
arXiv technical paper：https://arxiv.org/pdf/2606.26493

Comparison of Nemotron-Labs-TwoTower with similar competing products

Comparison dimension	Nemotron-Labs-TwoTower	LLaDA
Publishing organization	NVIDIA	MIT
Architecture design	Two-tower separation: frozen AR context tower + trainable diffusion denoising tower (connected via cross-attention)	Single-tower unification: a single Transformer simultaneously handles context encoding and masked denoising
Total parameter count	~60B (3B active, MoE sparse)	8B (dense)
Base model	Nemotron-3-Nano-30B-A3B（Mamba-Transformer MoE）	Self-developed Transformer
Training data	~2.1T tokens	~2T tokens
Baseline quality retention	98.7%(relative to the autoregressive baseline)	~95% (relative to an AR model of comparable scale)
Throughput improvement	2.42×（2×H100，wall-clock）	~1.5x (standard GPU environment)
Inference modes	Three：Mask Diffusion / Mock-AR / AR-only	Single: masked diffusion (random/semi-autoregressive sampling)
Attention mechanism	Context tower: causal attention; denoising tower: bidirectional block attention + cross-attention	Unified bidirectional attention + positional encoding processing
Core innovation	Role decoupling: avoids the performance bottleneck of a single network "serving two roles"	Simple and scalable: proves that diffusion models can scale to 8B and approach GPT-4 quality

Application scenarios of Nemotron-Labs-TwoTower

High-concurrency online services: the 2.42x throughput improvement makes it suitable for real-time text generation scenarios requiring low latency and high concurrency, such as search engines and intelligent customer service.
Long document generation: The parallel iterative nature of diffusion models makes them suitable for tasks requiring multiple refinement steps, such as long-text continuation, report generation, and code completion.
Flexible Multi-Mode Deployment: Switching between three inference modes allows enterprises to trade off cost and quality, using AR-only at the edge and Mask Diffusion in the cloud.
Commercial Product Development: The commercial-use license allows enterprises to integrate the model into commercial products such as writing assistants, marketing copy generators, and code assistance tools.
Research and Secondary Innovation: Open-source weights and paper details facilitate researchers' exploration of diffusion language model architectures, training strategies, and cross-modal extension directions.