What is Sonic
Sonic is an audio-driven portrait animation framework launched by Tencent and Zhejiang University, generating realistic facial expressions and movements based on global audio perception. Sonic is based on context-enhanced audio learning and a motion decoupling controller, which respectively extract long-term temporal audio knowledge within audio clips and independently control head and expression movements, enhancing local audio perception capabilities. Sonic uses a temporal-aware position offset fusion mechanism to extend local audio perception to the global level, solving the jitter and abrupt change issues in long video generation. Sonic outperforms existing state-of-the-art methods in video quality, lip synchronization accuracy, motion diversity, and temporal coherence, significantly improving the naturalness and coherence of portrait animation and supporting users' fine adjustments to the animation.
Sonic's main features
- Realistic lip synchronization: Precisely aligns audio with lip movements, ensuring that the spoken content is highly consistent with the mouth shapes.
- Rich expressions and head movements: Generates diverse and natural facial expressions and head movements, making the animation more vivid and expressive.
- Stable long-duration generation: When processing long videos, it can maintain stable output, avoiding jitter and abrupt changes and ensuring overall coherence.
- User adjustability: Supports users in adjusting and controlling head movement, expression intensity, and lip synchronization effects based on parameters, providing a high degree of customizability.
Sonic's technical principles
- Context-enhanced audio learning: Extracts long-term temporal audio knowledge within audio clips, converting information such as intonation and speaking rate in audio signals into prior knowledge for facial expressions and lip movements. The Whisper-Tiny model extracts audio features and, based on multi-scale understanding, combines the features with spatial cross-attention layers to guide the generation of spatial frames.
- Motion decoupling controller: Decouples head movement and expression movement, controlling them separately with independent parameters, enhancing the diversity and naturalness of the animation. Supports user-defined exaggerated movements, controlling the amplitude of head and expression movements based on adjusting motion-bucket parameters.
- Temporal-aware position offset fusion: Based on a temporal-aware sliding window strategy, it extends the local perception of audio clips to global perception, solving the jitter and abrupt change issues in long video generation. At each time step, the model starts processing the audio clip from a new position, gradually fusing global audio information to ensure the coherence of long videos.
- Global audio-driven: Sonic relies entirely on audio signals to drive animation generation, avoiding the dependence on visual signals (such as motion frames) in traditional methods and improving the naturalness and temporal consistency of generation. Audio signals serve as global signals, providing implicit prior information for facial expressions and head movements, making the generated animation more consistent with the audio content.
Sonic's experimental results
- Quantitative comparison：
- On the HDTF and CelebV-HQ datasets, Sonic outperforms existing SOTA methods on multiple evaluation metrics, including FID (Fréchet Inception Distance), FVD (Fréchet Video Distance), lip synchronization accuracy (Sync-C, Sync-D), and video smoothness (Smoothness).
- Sonic's FID and FVD scores are significantly lower than those of other methods, indicating that the videos it generates are of higher quality and have better consistency with real data.
- Qualitative comparison: Sonic can generate more natural and more diverse facial expressions and head movements, especially when handling complex backgrounds and portraits of different styles, showing stronger robustness.
Sonic's generation results
-
Comparison with open-source methods: Sonic can generate richer expressions that better match the audio, promoting more natural head movements.
-
Comparison with closed-source methods：
-
Comparison with EMO
-
Sonic performs better in the naturalness of facial expressions and the realism of eyeglass reflections.
-
-
-
-
-
In singing scenarios, Sonic demonstrates more precise pronunciation and more diverse movements.
-
-
-
-
Comparison with Jimeng：
-
In anime cases, Sonic's lip movements and appearance are closer to the original input, and it is accompanied by blinking movements.
-
-
-
-
-
In long video generation, Sonic is not limited by motion frames and avoids artifacts appearing at the end of the video.
-
-
Sonic's project addresses
- Project official website：https://jixiaozhong.github.io/Sonic/
- GitHub repository：https://github.com/jixiaozhong/Sonic
- arXiv technical paper：https://arxiv.org/pdf/2411.16331
- Online experience Demo：http://demo.sonic.jixiaozhong.online/
Sonic's application scenarios
- Virtual reality (VR): Generate realistic expressions and lip movements for virtual characters, enhancing immersion.
- Film and television production: Quickly generate lip movement and expression animations for characters, improving production efficiency.
- Online education: Convert teacher speech into vivid animations, enhancing learning interest.
- Game development: Generate natural expressions and movements for game characters, enhancing realism.
- Social media: Users can combine speech with photos to generate personalized animated videos for sharing.