What Is KeySync
KeySync is a high-resolution lip sync framework introduced by Imperial College London and the University of Wrocław, supporting the alignment of input audio with lip movements in videos. KeySync is implemented based on a two-stage framework: it first generates keyframes to capture key lip movements from the audio, and then generates smooth transition frames based on interpolation. KeySync introduces a new masking strategy that effectively reduces expression leakage from the input video and automatically handles occlusion using a video segmentation model. KeySync outperforms existing methods in visual quality, temporal coherence, and lip sync accuracy, making it suitable for practical applications such as automatic dubbing.
KeySync's Main Features
- High-Resolution Lip Sync: Generates high-definition (512×512) videos precisely aligned with the input audio, suitable for practical applications.
- Reduced Expression Leakage: Reduces expression leakage from the input video and improves the sync effect.
- Occlusion Handling: Automatically identifies and excludes occluders (such as hands, objects, etc.) during inference, ensuring the naturalness of the generated video.
- Improved Visual Quality: Performs excellently across multiple quantitative metrics and user studies, and the generated videos have higher clarity and coherence.
KeySync's Technical Principles
- Two-Stage Generation Framework：
- Keyframe Generation: First generates a set of sparse keyframes to capture the main lip movements in the audio, ensuring that each keyframe accurately reflects the speech content of the audio while preserving the person's identity information.
- Interpolation Generation: Interpolates between keyframes to generate smooth, temporally coherent intermediate frames, achieving smooth transitions in lip movements.
- Latent Diffusion Model: The latent diffusion model performs denoising operations in a compressed low-dimensional latent space to improve computational efficiency. It gradually removes noise and progressively transforms random noise into structured video data.
- Masking Strategy: Based on computed facial keypoints, a mask covering the lower face region is designed while retaining necessary contextual information to prevent expression leakage. During inference, it combines with a pretrained video segmentation model (such as SAM²) to automatically identify and exclude occluders, ensuring that the generated lip region blends naturally with the occluders.
- Audio and Video Alignment: Uses the HuBERT audio encoder to convert raw audio into feature representations, and embeds them into the video generation model based on an attention mechanism to ensure that the generated lip movements are precisely aligned with the audio.
- Loss Function: Combines latent space loss and pixel space loss (L2 loss) to optimize video generation quality and ensure that the generated lip region is aligned with the audio.
KeySync's Project Address
- Project Website：https://antonibigata.github.io/KeySync/
- GitHub Repository：https://github.com/antonibigata/keysync
- HuggingFace Model Library：https://huggingface.co/toninio19/keysync
- arXiv Technical Paper：https://arxiv.org/pdf/2505.00497
- Online Experience Demo：https://huggingface.co/spaces/toninio19/keysync-demo
KeySync's Application Scenarios
- Automatic Dubbing: Used for multilingual content production in film, television, advertising, and more, improving the alignment between dubbing and lip movements.
- Virtual Avatars: Generates synchronized lip movements for virtual characters, enhancing the realism of virtual avatars.
- Video Conferencing: Optimizes lip sync in remote communication and improves the user experience.
- Accessible Content: Help people with hearing impairments better understand video content.
- Content Repair: Repair or replace lip movements in videos to improve content quality.