What is JoyGen
JoyGen is an audio-driven 3D talking facevideo generationframework launched by JD Technology and the University of Hong Kong, focusing on achieving precise lip-audio synchronization and high-quality visual effects. JoyGen combines audio features and facial depth maps to drive the generation of lip movements, using a single-step UNet architecture for efficient video editing. During training, JoyGen uses a high-quality dataset containing 130 hours of Chinese videos, and validates its superior performance on the open-source HDTF dataset. Experimental results show that JoyGen achieves industry-leading levels in both lip-audio synchronization and visual quality, providing a new technical solution for the field of talking face video editing.
JoyGen's main features
- Lip-audio synchronization: Audio-driven lip movement generation technology ensures that the lip movements of the person in the video correspond precisely to the audio content.
- High-quality visual effects: The generated videos have realistic visual effects, including natural facial expressions and clear lip details.
- Video editing and optimization: Edit and optimize lip movements based on existing videos, without the need to regenerate the entire video.
- Multilingual support: Supports video generation in different languages such as Chinese and English, adapting to a variety of application scenarios.
JoyGen's technical principles
- Stage one：
- Audio-driven lip movement generation 3D reconstruction model: The 3D reconstruction model extracts identity coefficients from the input facial image, and the identity coefficients are used to describe the person's facial features.
- Audio-to-motion model: Based on the audio-to-motion model, the audio signal is converted into expression coefficients, and the expression coefficients are used to control lip movements.
- Depth map generation: Combine identity coefficients and expression coefficients to generate a 3D mesh of the face, and generate a facial depth map based on differentiable rendering technology for subsequent video synthesis.
- Stage two：
- Visual appearance synthesis single-step UNet architecture: Use a single-step UNet network to integrate audio features and depth map information into the video frame generation process. UNet maps the input image to a low-dimensional latent space based on the encoder, and combines audio features and depth map information to generate lip movements.
- Cross-attention mechanism: Audio features interact with image features based on the cross-attention mechanism, ensuring that the generated lip movements are highly consistent with the audio signal.
- Decoding and optimization: The generated latent representation is restored to image space based on the decoder to generate the final video frame. Optimization is performed in latent space and pixel space based on the L1 loss function to ensure the high quality and synchronization of the generated video.
- Dataset support: JoyGen uses a high-quality dataset containing 130 hours of Chinese videos for training, ensuring that the model can adapt to a variety of scenarios and language environments.
JoyGen's project address
- Project official website：https://joy-mm.github.io/JoyGen/
- GitHub repository：https://github.com/JOY-MM/JoyGen
- arXiv technical paper：https://arxiv.org/pdf/2501.01798
JoyGen's application scenarios
- Virtual anchors and livestreaming: Create virtual anchors to realize news broadcasting, e-commerce livestreaming, etc., generating realistic lip movements in real time based on input audio to enhance the audience experience.
- Animation production: In the field of animated film and television, quickly generate lip animations synchronized with voiceover, reducing animators' workload and improving production efficiency.
- Online education: Generate virtual teacher figures to achieve lip movements synchronized with teaching audio, making teaching videos more vivid and enhancing students' interest in learning.
- Video content creation: Help creators quickly generate high-quality talking-face videos, such as virtual character short dramas, funny videos, etc., enriching creative formats.
- Multilingual video generation: Supports multiple languages, quickly converting a video in one language into versions in other languages, with lip movements synchronized with the new language audio, facilitating international dissemination of content.