NeMo AutoModel is an open-source PyTorch DTensor-native SPMD training library under the NVIDIA NeMo Framework, designed to simplify and scale the training and fine-tuning of LLMs, VLMs, diffusion models, and retrieval models. It emphasizes flexibility, reproducibility, and scalability, supporting both small-scale experiments and large-scale multi-GPU, multi-node deployments, making it suitable for rapid research experimentation and production environments.
In terms of main capabilities, the library provides high-throughput scalable training: PyTorch DTensor-native SPMD training, where the same training script can scale from 1 GPU to large multi-node tasks by modifying the device mesh/configuration; composable parallelism including PyTorch-native FSDP2, HSDP, TP, CP, SP, and PP; optimized kernels using NVIDIA-oriented paths such as Transformer Engine, DeepEP, FlexAttn, TorchSDPA, fused attention, rotary embeddings, Triton, and optional kernel patches; MoE acceleration including MoE routing and DeepEP integration, as well as expert parallelism configurations used in DeepSeek, Qwen MoE, GPT-OSS, and Nemotron MoE benchmarks; support for FP8 and mixed precision (torchao and Transformer Engine), as well as MXFP8 MoE training on GB200; activation checkpointing to trade recomputation for lower activation memory; memory-efficient loss (Linear-Cut/fused linear cross-entropy) to avoid materializing full logits for the loss; sequence packing to reduce padding computation; FlashAttention packed sequence support to handle variable-length paths via per-document cu_seqlens; DCP support for PyTorch DCP and SafeTensors, sharded and consolidated layouts, merge/reshard tools, and Hugging Face-compatible outputs; asynchronous checkpointing with background writes to reduce I/O stalls; Dion and Muon optimizer integration; environment support for SLURM, interactive, SkyPilot, and Kubernetes launchers via SkyPilot.
In terms of algorithms, it supports pretraining (including DeepSeekV3), SFT, PEFT (LoRA, QLoRA), QAT (quantization-aware training), LLM knowledge distillation, VLM knowledge distillation (chunked KD lossand Qwen3.5-VL teacher-student recipe), speculative decoding training (EAGLE-1/2/3, P-EAGLE, DFlash, DFlash 2, DSpark).
In terms of usage, it adopts YAML-driven recipes, where any field can be overridden via CLI; it provides documentation, ready-to-use recipes, examples, and model override pages. The repository includes agent-oriented skills (skills/) to assist with common development tasks (such as running recipes, model onboarding, and development). Docker examples target the 26.10 container.
Applicable scenarios include pretraining, supervised fine-tuning, parameter-efficient fine-tuning, knowledge distillation, and speculative decoding training for LLM/VLM, as well as diffusion model and retrieval model training, covering everything from single-GPU experiments to large-scale multi-node deployments.
In terms of limitations, the materials do not provide specific pricing, costs, platform restrictions, or performance data; some features are marked as planned (🔜 Planned), and released features should be referenced in the release notes.
