What is EmoTalk3D
EmoTalk3D is a 3D digital human framework jointly launched by Huawei Noah's Ark Lab, Nanjing University, and Fudan University. The core of the technology lies in its ability to synthesize 3D talking heads with rich emotional expressions. EmoTalk3D can capture and reproduce human lip movements, facial expressions, and even more subtle emotional details such as wrinkles and other facial micro-movements during speech. Through a mapping framework called "Speech-to-Geometry-to-Appearance," EmoTalk3D achieves prediction from audio features to 3D geometry sequences, and then to the synthesis of 3D avatar appearance.
Main Features of EmoTalk3D
- Emotional expression synthesis: It can synthesize 3D avatar animations with corresponding emotional expressions based on input audio signals, including but not limited to various emotional states such as joy, sadness, and anger.
- Lip synchronization: Highly accurate lip movements synchronized with speech, so that the 3D avatar's lip movements match the actual pronunciation when speaking.
- Multi-view rendering: Supports rendering the 3D avatar from different angles, ensuring high quality and consistency when viewed from different perspectives.
- Dynamic detail capture: It can capture and reproduce facial micro-expressions and dynamic details during speech, such as wrinkles and subtle expression changes.
- Controllable emotion rendering: Users can control the emotional expression of the 3D avatar as needed, enabling real-time adjustment and control of emotions.
- High fidelity: Through advanced rendering technology, EmoTalk3D can generate high-resolution, highly realistic 3D avatars.
Technical Principles of EmoTalk3D
- Dataset construction (EmoTalk3D Dataset)：Multi-view video data was collected, including emotion annotations and 3D facial geometry information for each frame.The dataset comes from multiple subjects, with each subject recorded in multi-view videos under different emotional states.
- Audio feature extraction：A pretrained HuBERT model is used as the audio encoder to convert input speech into audio features.Emotion labels are extracted from audio features through an emotion extractor.
- Speech-to-Geometry Network (S2GNet)：Audio features and emotion labels are used as input to predict dynamic 3D point cloud sequences. Based onGated Recurrent Unit (GRU) as the core architecture, 4D mesh sequences are generated.
- Mapping from 3D geometry to appearance (Geometry-to-Appearance)：Based on the predicted 4D point cloud, the Geometry-to-Appearance Network (G2ANet) is used to synthesize the appearance of the 3D avatar.The appearance is decomposed into canonical Gaussians (static appearance) and dynamic Gaussians (wrinkles, shadows, etc. caused by facial motion).
- 4D Gaussian model：3D Gaussian Splatting technology is used to represent the appearance of the 3D avatar.Each 3D Gaussian is parameterized by position, scale, rotation, and opacity.
- Dynamic detail synthesis：Dynamic details such as wrinkles and subtle expression changes are predicted through FeatureNet and RotationNet networks.
- Head completeness：For non-facial regions such as hair, neck, and shoulders, optimization algorithms are used to construct them starting from uniformly distributed points.
- Rendering module：Dynamic Gaussians and canonical Gaussians are fused to render 3D avatar animations with free viewpoints.
- Emotion control：Control the emotional expression of generated avatars through manual setting of emotion labels and changes in time series.
EmoTalk3D project address
-
Project website：https://nju3dv.github.io/projects/EmoTalk3D
-
arXivTechnical paper：https://arxiv.org/abs/2408.00297
Application scenarios of EmoTalk3D
- Virtual assistants and customer service: As intelligent customer service or virtual assistant roles, provide more natural and emotionally rich interactive experiences.
- Film and video production: Generate realistic characters and animations in movies, TV, and video games to improve the visual experience.
- Virtual reality (VR) and augmented reality (AR): Provide immersive experiences in VR and AR applications, enabling more realistic interaction with users.
- Social media and live streaming: Users can use EmoTalk3D to create and customize their own 3D avatars on social media platforms or in live streams.
- Advertising and marketing: Create attractive 3D characters for advertising or brand promotion.