What is LatentSync
LatentSync is an end-to-end lip sync framework jointly launched by ByteDance and Beijing Jiaotong University. It is based on an audio-conditioned latent diffusion model and does not require any intermediate 3D representations or 2D facial landmarks. LatentSync uses the powerful generative capabilities of Stable Diffusion to capture complex audio-visual correlations and generate dynamic, realistic talking videos. To address the temporal consistency problem caused by inconsistent diffusion processes across different frames in diffusion models, it introduces the Temporal REPresentation Alignment (TREPA) method, which uses large-scale self-supervised video models to extract temporal representations, enhancing the temporal consistency between generated frames and real frames while maintaining lip sync accuracy. Based on comprehensive experimental research, LatentSync solves the convergence problem of SyncNet and improves the accuracy of lip sync.
Main Features of LatentSync
- Lip Sync Generation: Generates lip movements that match the input audio, synchronizing the lips of the person in the video with the audio. It is suitable for scenarios such as dubbing and virtual avatars.
- High-Resolution Video Generation: Generates high-resolution videos, overcoming the high hardware requirements of traditional diffusion models when performing diffusion in pixel space.
- Dynamic and Realistic Effects: The generated videos have dynamic and realistic effects, can capture subtle expressions related to emotional tone, and make the person's speech more natural and vivid.
- Enhanced Temporal Consistency: Based on the Temporal REPresentation Alignment (TREPA) method, it improves the temporal consistency of generated videos, reduces video flickering, and makes video playback smoother.
Technical Principles of LatentSync
- Audio-Conditioned Latent Diffusion Model: Conditioned on audio, it uses a latent diffusion model to directly perform modeling in latent space, without going through pixel-space diffusion or a two-stage generation process. The latent diffusion model can better capture the complex correlations between audio and vision and generate high-quality lip-synced videos.
- End-to-End Framework: Based on an end-to-end framework structure, it integrates processes such as audio feature extraction, latent representation generation, and lip sync generation into a unified model, simplifying intermediate steps and improving generation efficiency and accuracy.
- Temporal REPresentation Alignment (TREPA): It introduces the TREPA method, uses the large-scale self-supervised video model VideoMAE-v2 to extract temporal representations, and calculates the distance between the temporal representations of generated consecutive frames and real consecutive frames as an additional loss to enhance the temporal consistency of generated videos.
- SyncNet Supervision: During training, the pre-trained SyncNet is used to supervise the generated videos to ensure that the generated videos have good lip sync effects. A SyncNet loss is added in pixel space, allowing the model to better learn the correspondence between audio and lip movements.
Project Address of LatentSync
- GitHub Repository：https://github.com/bytedance/LatentSync
- arXiv Technical Paper：https://arxiv.org/pdf/2412.09262
Application Scenarios of LatentSync
- Film and Television Post-Production: During movie dubbing, it automatically generates matching lip animations based on the dubbed audio, improving production efficiency and maintaining the consistency of the character's image.
- Education: In online English classes, teachers convert speech into lip-synced videos to help students learn pronunciation more accurately.
- Advertising Video Production: In car advertisements, it generates lip-synced videos for virtual spokespersons, making the advertising copy expression more natural and enhancing the appeal of the advertisement.
- Remote Meetings: Real-time generation of lip-synced video in cross-national remote meetings, solving the audio-video desynchronization problem caused by network latency and improving communication effectiveness.
- Game Development: In RPG games, making NPCs' lip movements synchronize with their speech during dialogue, enhancing game immersion and the character interaction experience.