DICE-Talk - An Emotional Dynamic Portrait Generation Framework Jointly Launched by Fudan University and Tencent Youtu

What is DICE-Talk

DICE-Talk is a novel emotional dynamic portrait generation framework jointly launched by Fudan University and the Tencent Youtu Lab. It supports generating dynamic portrait videos with vivid emotional expression while maintaining identity consistency. DICE-Talk introduces an emotion correlation enhancement module that captures the relationships between different emotions based on an emotion bank, improving the accuracy and diversity of emotion generation. The framework designs an emotion discrimination objective, ensuring emotion consistency during the generation process based on emotion classification. Experiments on the MEAD and HDTF datasets show that DICE-Talk outperforms existing technologies in emotion accuracy, lip sync, and visual quality.

Main Features of DICE-Talk

• Emotional Dynamic Portrait Generation: Generates dynamic portrait videos with specific emotional expressions based on input audio and reference images.
• Identity Preservation: While generating emotional videos, it preserves the identity features of the input reference image, avoiding leakage or confusion of identity information.
• High-Quality Video Generation: The generated videos achieve a high level in visual quality, lip synchronization, and emotional expression.
• Generalization Ability: It can adapt to unseen identity and emotion combinations, with good generalization ability.
• User Control: Users input specific emotion targets to control the emotional expression of the generated video, achieving a high degree of user customization.
• Multimodal Input: Supports multiple input modalities, including audio, video, and reference images.

Technical Principles of DICE-Talk

• Decoupling Identity and Emotion: Based on a cross-modal attention mechanism, it jointly models audio and visual emotional cues, representing emotions as identity-independent Gaussian distributions. It trains an emotion embedder with contrastive learning (such as InfoNCE loss) to ensure that features of the same emotion cluster in the embedding space, while features of different emotions are dispersed.
• Emotion Correlation Enhancement: The emotion bank is a learnable module that stores feature representations of multiple emotions. It uses vector quantization and attention-based feature aggregation to learn the relationships between emotions. The emotion bank stores the features of a single emotion and learns the associations between emotions, helping the model better generate other emotions.
• Emotion Discrimination Objective: During the generation process of the diffusion model, an emotion discriminator ensures the emotion consistency of the generated video. The emotion discriminator is jointly trained with the diffusion model to ensure that the generated video is consistent with the target emotion in emotional expression, while maintaining visual quality and lip synchronization.
• Diffusion Model Framework: Starting from Gaussian noise, it gradually denoises to generate the target video. Based on a variational autoencoder (VAE), video frames are mapped to a latent space, Gaussian noise is gradually introduced in the latent space, and the diffusion model gradually removes the noise to generate the target video. During denoising, the diffusion model combines reference images, audio features, and emotion features based on a cross-modal attention mechanism to guide video generation.

DICE-Talk Project Address

• Project Official Website：https://toto222.github.io/DICE-Talk/
• GitHub Repository：https://github.com/toto222/DICE-Talk
• arXiv Technical Paper：https://arxiv.org/pdf/2504.18087

Application Scenarios of DICE-Talk

• Digital Humans and Virtual Assistants: Endows digital humans and virtual assistants with rich emotional expression, making interactions with users more natural and vivid, and improving user experience.
• Film and Television Production: In film and television special effects and animation production, quickly generate dynamic portraits with specific emotions, improve production efficiency, and reduce production costs.
• Virtual Reality and Augmented Reality: In VR/AR applications, generate virtual characters that interact with users' emotions, enhancing immersion and emotional resonance.
• Online Education and Training: Create teaching videos with emotional feedback, making learning content more vivid and interesting, and improving learning outcomes.
• Mental Health Support: Develop emotional virtual characters for use in psychotherapy and emotional support, helping users better express and understand emotions.