Molt is an agentic-first, research-oriented reinforcement learning framework released by NVIDIA-NeMo, designed natively with PyTorch and HuggingFace, with approximately 9.2K lines of RL code in total. Its positioning is the smallest possible native PyTorch / HuggingFace technology stack, used to drive 1T-level fully asynchronous, multimodal, multi-turn agentic reinforcement learning, and it can scale to frontier MoE scale on vLLM.
[Purpose and Design Philosophy] Molt is agent-centric: the agent is the program, the trainer is a single actor, and the reward is any Python code you write in an Env or ChatAgent, and can be a scorer, multi-turn tool, VLM environment, or LLM-as-judge. The framework is carried by three components: Ray handles placement and asynchronous queues, vLLM handles rollout, and NVIDIA AutoModel + FSDP2 handles pure PyTorch training. The entire runtime is "three boxes + one asynchronous loop." The contract is token-first: token ids, logprobs, action ranges, rewards, and multimodal tensors remain aligned from rollout to training. Anything computable in Python can serve as a valid reward, including calling an LLM-as-judge through the same batch of vLLM engines that drive rollout.
[Main Capabilities] In terms of training and runtime, it supports SFT (molt.cli.train_sft) and vLLM-based online RL (molt.cli.train_rl_ray); the runtime includes Ray placement, asynchronous rollout queues, vLLM engines, and partial rollout synchronization; in terms of model scale, it supports TP / EP / CP and MoE natively through AutoModel + FSDP2, for example DeepSeek-V3 runs under --fsdp.ep_size 256; the model backend uses NVIDIA AutoModel as the primary path, supports native CP / EP / TP, custom MoE+EP parallelizers, and TE fused attention, while the HF transformers path is a non-preferred fallback (supporting only text + flash_attention_2 + packing, no CP / EP / TP). The optimizer defaults to adam, and CPU offload can be enabled for the largest actor (--fsdp.offload optimizer); muon is experimental (via Dion's Newton–Schulz, using Muon for 2D weights and grouped MoE experts, and AdamW for embedding / head / norm), and has been run under distributed (FSDP / EP) settings, but has not yet shown a stable advantage over adam, so adam remains the recommended default.
On the agent and reward side, estimators support reinforce, reinforce_baseline, rloo, grpo, dr_grpo, gae (PPO), and on_policy_distill. The PPO critic adds a value model via --algo.advantage.estimator gae, has an independent Ray group (CriticModelActor), by default is co-located with the actor's GPU or can be separated, uses GAE advantages (--algo.advantage.lam) and a clipped value loss (--critic.value_clip), an independent optimizer/learning rate (--critic.adam.lr), and a recoverable _critic checkpoint, is based on NeMoAutoModelForCausalLM and a scalar value head, and preserves the native TP / EP / CP paths. For distillation, on-policy distillation is supported: per-token reverse KL against a frozen teacher, enabled via --algo.advantage.estimator on_policy_distill + --ref.model_name_or_path. IS correction is used to correct training/rollout logprob mismatch for off-policy / asynchronous rollout, supporting is_correction_level {off,token,seq,geo} × is_correction_mode {mask,clip,trunc}, covering TIS, IcePop, seq-mask-tis. On the KL side, optional reference workers can be enabled when --algo.kl.init_coef > 0 (the reference also serves as the distillation teacher).
As of 2026-09, Molt supports FlashREINFORCE, i.e. critic-free single-rollout RL, and training remains stable after more than 6,000 steps.
[Usage] Installation offers two paths: the container is recommended; dockerfile/Dockerfile includes a complete CUDA-13 stack (torch 2.13, vLLM, TransformerEngine, flash-attn, mamba, DeepEP, NVIDIA AutoModel), is compatible with A100 / H100 / H200 / B200 / GB200, and can run SFT and RL directly; since 0.1.9 the tag is multi-arch (amd64 + arm64), so the same pull can be used on x86 and Grace-Blackwell hosts. Local installation is intended for development outside the container, pulls the git-pinned AutoModel verified by this repository (see setup.py, AUTOMODEL), requires CUDA 13 (torch==2.13.0+cu130), and is unavailable in CUDA-12 environments; if the host driver is lower than 580, the container should be used (it has a built-in CUDA forward-compatibility layer and enables it automatically). The molt-rl package on PyPI lags behind the pin because it cannot carry the git-pinned AutoModel, and is currently not a supported installation path. The optional backend AutoModel-Slim is a pinned AutoModel copy maintained by molt itself, with the same package name (nemo_automodel), import path, and API, so no molt code changes are needed when switching; it preserves the model families and training stack used by molt training, and reduces files and code volume from 758 files / 282k lines to 252 / about 97k; the default remains upstream NVIDIA AutoModel.
On the agent contract side, each RL run points to a Python module, which must export AgentRunner, with two optional paths: Env (the framework controls the LLM loop, Gymnasium-style step/reset, the framework drives vLLM, tokenization, multimodal accounting, and per-turn budget, step() returns a Result, and the framework chains turns until terminated or truncated); ChatAgent (controls the loop itself via the OpenAI or Anthropic SDK). The server supports both OpenAI /v1/chat/completions and Anthropic /v1/messages, and both are decoded into the same exact token accumulation; when a request rewrites the prefix rather than extending it, the server automatically seals the current segment and starts a new exact-token segment from the re-templated conversation, so a single rollout can produce multiple segment trajectories that share the reward and rollout_id, group baselines (GRPO/RLOO, etc.) will deduplicate them into one reward per rollout, while each segment still contributes its own generated tokens for the policy gradient, with no agent-side changes needed.
For length penalties, two optional penalties are provided, off by default and mutually exclusive: --reward.overlong_buffer_len B linearly increases the penalty in DAPO style over the last B tokens of --data.max_len up to --reward.overlong_penalty_factor (default 1); --reward.stop_properly_penalty_coef C multiplies the reward of a truncated rollout by C (C within [0,1]), or replaces it with C when C is negative. Truncation covers the generation length limit, context exhaustion, and the agent turn cap. Both are measured by the rollout's peak segment, not summed.
Reference launch scripts are located in examples/scripts/, all running end-to-end on the AutoModel + FSDP2 backend. The geo3k VLM scripts (rl_qwen3_6_35b.sh / sft_qwen3_6_35b.sh) automatically prepare the dataset via examples/python/utils/prepare_geo3k.py on first run, or you can pre-provision it manually or point to your own PROMPT_DATASET / EVAL_DATASET. For a multi-turn Python tool environment example, see examples/python/agents/geo3k.py, the model emits a <tool_call> invoking python_executor(code=...), the environment runs the code in a sandbox subprocess and returns stdout as <tool_response>, the loop runs at most MAX_AGENT_TURNS (default 5 for agent; set to 4 in the Qwen3.6 quick start, and 10 on Slurm), and the final <answer>ANSWER</answer> (or the legacy \boxed{ANSWER}) is compared with the ground truth as the reward.
[Applicable scenarios and limitations] Molt is aimed at agentic reinforcement learning research, pursuing speed in agentic research on large-scale frontier MoE, rather than ecosystem breadth. It is suitable for research scenarios that require fully asynchronous agentic RL on vLLM, multi-turn and multimodal agent training, the full SFT-to-RL pipeline, and a highly readable PyTorch-native stack. In terms of limitations: local installation requires CUDA 13, and CUDA-12 environments are unavailable; the muon optimizer is still experimental and has not shown a stable advantage over adam; the HF transformers fallback path does not support CP / EP / TP; the molt-rl package on PyPI is currently not a supported installation path. Specific pricing and cost information is not yet provided.
