What Is MOVA
MOVA (MOSS Video and Audio) is China's first high-performance open-source end-to-end audio-video generation model, jointly launched by the OpenMOSS team at Shanghai Chuangzhi Institute and Mosi Intelligence (MOSI). The model breaks through the traditional video "silent" limitation, adopting a heterogeneous dual-tower architecture and bidirectional bridging module to achieve native cross-modal interaction. The model has 32 billion parameters (MoE architecture, with 18 billion activated during inference), and can synchronously generate videos up to 8 seconds long at 720p resolution along with accompanying audio, excelling in cinematic lip synchronization and environmental sound effect matching.
MOVA's Main Features
-
End-to-End Audio-Video Generation: The model can synchronously output video and accompanying audio in one go, saying goodbye to "mute videos."
-
Dual-Mode Driven Generation: Supports image + text or pure text input, flexibly controlling the generated content.
-
Cinematic Lip Synchronization: The model can precisely match the mouth movements and speech of characters when speaking, supporting Chinese and English multi-character dialogue.
-
Intelligent Environmental Sound Effects: Automatically synthesizes matching background music, action sounds, and ambient sounds based on the scene in the frame.
-
Video Text Rendering: The model can generate clear and readable dynamic text content at specified positions in the frame.
-
High-Resolution Output: The model supports generation of audiovisual clips up to 720p resolution and 8 seconds in duration.
MOVA's Technical Principles
- Heterogeneous Dual-Tower Architecture: The model uses a 14B video diffusion model and a 1.3B audio diffusion model to process visual and auditory information separately, and through a bidirectional bridging module achieves deep cross-attention fusion of the two layers' hidden states, allowing the image generation process to perceive the rhythm of sound throughout.
- Cross-Modal Temporal Alignment: The sampling densities of video and audio differ greatly. The Aligned ROPE mechanism uses precise scaling ratio mapping to unify the tokens of the two modalities into the same physical time coordinate system, fundamentally eliminating audio-video desynchronization.
- Progressive Training Strategy: The model is trained in three stages from coarse to fine. It first uses 360p low resolution to let the randomly initialized bridging module quickly learn audio-video alignment, gradually improving alignment stability, and finally extends to 720p high resolution for image quality refinement.
- Dual CFG Inference: In view of the fact that joint audio-video generation has two control sources, text instructions and modal bridging, it supports independently adjusting their guidance weights, preserving image quality in general scenes and strengthening lip precision in dialogue scenes.
MOVA's Project Address
- Project Official Website：https://mosi.cn/models/mova
- GitHub Repository：https://github.com/OpenMOSS/MOVA
- HuggingFace Model Library：https://huggingface.co/collections/OpenMOSS-Team/mova
MOVA's Application Scenarios
- Film and Television Production: Rapidly generate storyboard previews and dubbing samples, reducing pre-production costs and accelerating creative validation.
- Short Video Creation: Provide creators with high-quality story material with sound effects, improving output efficiency and enriching content formats.
- Game Development: Automatically generate cutscenes and character dialogue, achieving an immersive experience with synchronized audio and visuals and shortening the development cycle.
- Education and Training: Produce multilingual teaching videos with precise lip synchronization, supporting global content adaptation and improving learning outcomes.
- E-commerce Marketing: Produce product showcase videos with narration and background music, accelerating marketing content iteration and enhancing conversion capabilities.