What is MimicTalk
MimicTalk is jointly developed and launched by Zhejiang University and ByteDance. Based on NeRF (Neural Radiance Field) technology, it can train a personalized and expressive 3D talking face model in an extremely short time, requiring only 15 minutes. MimicTalk improves training efficiency and, based on an efficient fine-tuning strategy and a facial motion generation model with in-context learning capability, significantly enhances the quality of generated videos. MimicTalk's code is further optimized based on the previous Real3D-Portrait project, and supports audio-driven generation of talking portraits of specific people, making digital human videos highly similar to real people in sensory perception.
MimicTalk's main features
- Fast personalized training: MimicTalk can complete adaptation and training for a new identity within 15 minutes, greatly improving training efficiency compared with traditional methods.
- High-quality video generation: Based on fine-tuning and optimization, the videos generated by MimicTalk surpass previous technologies in visual quality.
- Enhanced expressiveness: MimicTalk can capture and imitate the target person's dynamic speaking style, making the generated videos more vivid and expressive.
- In-context learning: The model can learn the target person's speaking style from context, improving the naturalness and realism of facial motions.
- Audio-driven: Supports using audio input to drive a specific person's 3D talking portrait, achieving synchronization from audio to facial motions.
MimicTalk's technical principles
- Person-Agnostic 3D Face Generation Model：
- As the base model, it is a pretrained general-purpose 3D face generation model capable of processing single-image input.
- The model can generate realistic 3D faces for different person identities.
- Static-Dynamic Hybrid Adaptation Pipeline：
- The hybrid adaptation pipeline helps the model learn the static appearance of a specific identity (such as facial geometry and texture details) and dynamic features (such as facial expressions and muscle movements).
- Based on optimizing 3D face representations (tri-plane) and using Low-Rank Adaptation (LoRA) technology, the model can quickly adapt to new identities.
- In-Context Stylized Audio-to-Motion Model, ICS-A2M：
- The model is responsible for generating facial motions that match the target person's speaking style.
- Based on in-context learning, it imitates the speaking style in the reference video without requiring explicit style representation.
- Flow Matching Model：
- An advanced generative model used to generate expressive facial motions, predicting the velocity field of data points and guiding data points to move from a simple prior distribution to the target distribution.
- During training, the Conditional Flow Matching (CFM) objective is used to optimize the accuracy of the model's predictions.
- Inference Process：
- In the inference stage, MimicTalk uses the ICS-A2M model and a personalized renderer to generate high-quality talking face videos.
- The inference process involves combining audio input with the target person's reference video to produce facial motions that imitate a specific speaking style.
- Data and training efficiency: MimicTalk's design focuses on sample efficiency and training efficiency, and can complete adaptation to a new identity with a small amount of data in an extremely short time.
MimicTalk project address
- Project official website：mimictalk.github.io
- GitHub repository：https://github.com/yerfor/MimicTalk
- arXiv technical paper：https://arxiv.org/pdf/2410.06734
MimicTalk application scenarios
- Virtual anchors and digital humans: In news broadcasting, entertainment programs, and online live streaming, use MimicTalk technology to create virtual anchors, providing a more natural and engaging viewing experience.
- Video conferencing and remote collaboration: In remote work and online meetings, MimicTalk provides users with personalized virtual avatars, increasing interactivity and immersion.
- Virtual reality (VR) and augmented reality (AR): In VR and AR applications, MimicTalk generates realistic virtual characters, enhancing the gaming experience and interaction quality.
- Social media and entertainment: Users create their own virtual avatars, share them on social media, or interact with others in the virtual world.
- Customer service and chatbots: Based on MimicTalk technology, develop more humanized customer service robots, providing a more natural and friendly customer experience.