LMMs-Engine is a streamlined, efficient framework for training unified multimodal models, designed to support large-scale multimodal model training. It is maintained by EvolvingLMMs-Lab, uses the Apache 2.0 license, and its code is hosted on GitHub.
Purpose LMMs-Engine is primarily used for pretraining and fine-tuning multimodal models, supporting various modality combinations such as vision-language, audio-language, and diffusion models. It is suitable for research and large-scale training scenarios, providing a one-stop solution from dataset preparation to model training.
Main Capabilities
- Supports more than 20 model architectures, including vision-language and multimodal models such as Qwen2.5-VL, Qwen3-VL, Qwen3-VL MoE, Qwen2.5-Omni, Qwen3-Omni MoE, LLaVA-OneVision, Bagel, and Aero, as well as diffusion and generative models such as dLLM, WanVideo, SiT, and RAE-SigLip, and also Qwen-series language models and linear attention models.
- Provides multiple distributed training optimizations: FSDP2 (DTensor-based sharding of parameters, gradients, and optimizer states), Ulysses sequence parallelism (splits the sequence dimension across GPUs, supporting ultra-long context), tensor parallelism, expert parallelism (for MoE models), and combinations of multi-dimensional parallelism.
- Memory and computation optimizations: Flash Attention + Unpadding, native sparse attention (NSA), Liger kernels (Triton fused kernels, reducing memory usage), Monkey Patching system (runtime kernel injection), sequence packing (first-fit bin packing, improving MFU).
- Advanced optimizer: Muon optimizer (Newton-Schulz orthogonalization, Triton kernels, distributed via DTensor, selectively applied to 2D parameters).
- Supports streaming datasets (IterableDataset), suitable for trillion-token pretraining.
- Provides Docker images with PyTorch, CUDA, FlashAttention, and training dependencies preinstalled.
- Supports launching training via torchrun or Accelerate.
- Provides detailed MFU reference metrics and example configurations.
Usage
- Dataset preparation: process the dataset into OpenAI chat format (JSONL/JSON/Arrow/CSV); example datasets can be downloaded using the hf download command.
- Configure the dataset YAML (optional for a single data source), specifying the path, data folder, and data type.
- Configure training: refer to examples/qwen3_vl/example_config.yaml or the model-specific configurations under examples/.
- Launch training: torchrun (native PyTorch) is recommended, or use Accelerate.
- Each run.sh file contains detailed setup instructions, prerequisites, and configuration options.
Applicable Scenarios
- Vision-language pretraining: train Qwen-VL, LLaVA, etc. on large multimodal datasets.
- Video understanding: train AERO on 3D video data.
- Diffusion models: train DLLM, SiT, and WanVideo for generative tasks.
- Representation learning: train RAE for visual representation.
- Language model pretraining: train DGN and Qwen using the Muon optimizer.
- Multimodal fine-tuning: use sequence packing for efficient SFT.
Limitations
- Native sparse attention (NSA) currently only supports the BAGEL model.
- Linear attention models require FLA to be installed first.
- The Muon optimizer by default excludes modules containing specific substrings (such as emb, norm, lm_head, etc.) as well as parameters whose dimension is not equal to 2.
- Some features may require specific dependencies and configurations; please refer to the official documentation for details.
Documentation and Support Detailed documentation is provided, including dataset preparation, training guides, design principles, and API references. The project welcomes contributions and provides design principles to guide code style. Thanks to multiple open-source projects for their support.
