What VeOmni Is
VeOmni is an omni-modal distributed training framework open-sourced by ByteDance's Seed team, designed based on PyTorch. VeOmni is model-centric, decoupling distributed parallel logic from model computation, supporting flexible combination of multiple parallel strategies (such as FSDP, SP, EP), and can efficiently scale to ultra-long sequences and large-scale MoE models. VeOmni provides a lightweight omni-modal interface, simplifying the integration of multimodal codecs, and integrates optimization techniques such as dynamic batching and efficient operators, greatly improving training efficiency and stability. VeOmni has been applied in multiple frontier projects, facilitating research and development of omni-modal large models.
VeOmni's Main Features
- Supports Omni-Modal Model Training: VeOmni can support model training for any modality (such as text, images, audio, video, etc.), applicable to a variety of tasks from single-modal to omni-modal.
- Efficient Distributed Training: Supports flexible combination of multiple parallel strategies (such as FSDP, SP, EP), and can efficiently scale to large-scale GPU clusters.
- Ultra-Long Sequence Support: Supports ultra-long sequence training of up to 192K, suitable for processing complex modal data such as high-definition images and long videos.
- Lightweight Interface and Ease of Use: Supports rapid integration of multimodal codecs, simplifying the model development process.
- System-Level Optimization: Integrates technologies such as dynamic batching, efficient operators, recomputation and memory optimization, and ByteCheckpoint, improving training efficiency and stability.
- Training Stability: Performs excellently in complex multimodal tasks, supports stable convergence, and is applicable to various scenarios in practical applications.
- Flexible Model Extension: Supports multiple model architectures (such as MoE, Transformer, etc.), allowing users to customize model components to meet different research and development needs.
VeOmni's Technical Principles
- Model-System Decoupling: VeOmni separates model definition from distributed training logic, making model code and parallel strategies completely decoupled. Users can configure parallel strategies through high-level APIs without modifying model code.
- Distributed Parallel Strategies: Shards model parameters, gradients, and optimizer states across multiple devices, significantly reducing the memory requirements of a single GPU. Based on partitioning activation tensors and optimizing communication, supports ultra-long sequence training. Shards experts in MoE models across multiple devices, improving the training efficiency of MoE models. Designs parallel_state based on DeviceMesh, simplifying the management of n-D parallel strategies and allowing free combination of different parallel strategies.
- Lightweight Omni-Modal Interface: Uses the HuggingFace interface specification, supporting users to quickly integrate multimodal codecs by implementing unified interface functions (such as lm_encode, lm_generate, etc.).
- System-Level Optimization: Integrates multiple optimization technologies, such as dynamic batching, efficient operators, recomputation and memory optimization, and ByteCheckpoint, etc., comprehensively improving training efficiency and stability.
VeOmni's Project Address
- GitHub Repository：https://github.com/ByteDance-Seed/VeOmni
- arXiv Technical Paper：https://arxiv.org/pdf/2508.02317
VeOmni's Application Scenarios
- Multimodal Content Generation: Generates images or videos based on text descriptions, or generates descriptive text for images or videos, widely used in creative design and content creation.
- Multimodal Understanding and Question Answering: Answer visual questions by combining images and text, or handle complex question-answering tasks involving multiple modalities, enhancing the intelligent interaction experience.
- Multimodal intelligent agents: Support the development of virtual assistants and multimodal robots that can interact with users and perform tasks through voice, text, and visual information.
- Content creation and editing: Generate creative design elements based on text descriptions, assist with content moderation, and improve the efficiency of content creation and editing.
- Education and training: Provide virtual training to enhance the interactivity and effectiveness of education and training.