What Is OmniSync
OmniSync is a general-purposelip-syncframework jointly launched by Renmin University of China, Kuaishou Technology, and Tsinghua University. It is based on Diffusion Transformers to achieve precise synchronization of character lip movements with speech in videos. OmniSync directly edits video frames based on a mask-free training paradigm, requiring no reference frames or explicit masks, and supports infinite-length inference while maintaining natural facial dynamics and identity consistency. OmniSync introduces flow-matching-based progressive noise initialization and a dynamic spatiotemporal classifier-free guidance (DS-CFG) mechanism to address the problem of weak audio signals and ensure precise lip synchronization. OmniSync establishes the AIGC-LipSync benchmark to evaluate lip-sync performance in AI-generated videos.
Main Features of OmniSync
- Mask-Free Training: Directly edits video frames without reference frames or masks, and supports infinite-length inference.
- Identity Preservation: Ensures head pose and identity consistency while precisely modifying the mouth region.
- Enhanced Audio Conditioning: Based on a dynamic spatiotemporal guidance mechanism, it addresses the problem of weak audio signals.
- General Compatibility: Suitable for stylized characters, non-human entities, and AI-generated content.
- Infinite-Length Inference: Maintains natural facial dynamics and temporal consistency.
- Occlusion Robustness: Maintains high-quality lip synchronization under complex conditions such as facial occlusion.
Technical Principles of OmniSync
- Mask-Free Training Paradigm: Performs direct cross-frame editing based on Diffusion Transformers, without explicit masks or reference frames. Based on iterative denoising to learn the mapping function, it introduces a timestep-dependent sampling strategy, using different datasets at different denoising stages to ensure stable learning.
- Progressive Noise Initialization: Based on Flow Matching, it injects control noise into the original frames and performs only the final denoising step, maintaining spatial consistency and supporting precise modification of the mouth region, effectively addressing the problems of pose inconsistency and identity drift.
- Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG): Provides fine-grained control over audio influence, balancing audio conditioning strength based on spatiotemporal adaptive guidance. Spatial adaptive guidance uses a Gaussian-weighted spatial guidance matrix, concentrating guidance strength on the mouth region. Temporal adaptive guidance gradually reduces guidance strength as the denoising process progresses, ensuring strong guidance in the early and middle diffusion stages and reducing interference when refining details in the later stage.
OmniSync Project Address
- Project Official Website：https://ziqiaopeng.github.io/OmniSync/
- arXiv Technical Paper：https://arxiv.org/pdf/2505.21448
Application Scenarios of OmniSync
- Film and Television Dubbing: Achieves precise matching between character lip movements and dubbing.
- Virtual Reality: Provides realistic lip synchronization for virtual characters, enhancing immersion.
- AI Content Generation: Improves the naturalness of lip synchronization in AI-generated videos.
- Video Conferencing: Improves the lip-sync effect in remote communication.
- Game Development: Enhances the lip performance of game characters and improves interactivity.