EMO - Alibaba's AI Portrait Video Generation Framework

What is EMO

EMO (Emote Portrait Alive) is a framework developed by researchers at Alibaba Group's Institute for Intelligent Computing. It is an audio-driven AI portrait video generation system that can generate videos with expressive facial expressions and various head poses from a single reference image and speech audio input. The system is able to capture the nuances of human expressions and the diversity of individual facial styles, thereby generating highly realistic and expressive animations.

EMO Official Website Entry

• Official Project Homepage:https://humanaigc.github.io/emote-portrait-alive/
• arXiv Research Paper:https://arxiv.org/abs/2402.17485
• GitHub：https://github.com/HumanAIGC/EMO(Model and source code pending open source)

Main Features of EMO

• Audio-Driven Video Generation: EMO can directly generate videos based on input audio (such as speaking or singing) without relying on pre-recorded video clips or 3D facial models.
• High Expressiveness and Realism: Videos generated by EMO are highly expressive, able to capture and reproduce the nuances of human facial expressions, including subtle micro-expressions, as well as head movements that match the rhythm of the audio.
• Seamless Frame Transitions: EMO ensures natural and smooth transitions between video frames, avoiding facial distortion or inter-frame jitter, thereby improving the overall video quality.
• Identity Preservation: Through the FrameEncoding module, EMO is able to maintain character identity consistency during video generation, ensuring the character's appearance remains consistent with the input reference image.
• Stable Control Mechanisms: EMO employs stable control mechanisms such as speed controller and facial region controller to enhance stability during video generation and avoid issues such as video collapse.
• Flexible Video Duration: EMO can generate videos of any duration based on the length of the input audio, providing users with flexible creative space.
• Cross-Language and Cross-Style: EMO's training dataset covers multiple languages and styles, including Chinese and English, as well as realism, anime, and 3D styles, enabling EMO to adapt to different cultures and artistic styles.

How EMO Works

1. Input Preparation: The user provides a reference image (usually a static portrait of the target character) and corresponding audio input (such as speaking or singing voice). These inputs serve as the basis for generating the video.
2. Feature Extraction: ReferenceNet is used to extract features from the reference image. ReferenceNet is a network with a structure similar to the main network (Backbone Network), focused on extracting detailed features from the input image.
3. Audio Processing: The audio input is processed through a pre-trained audio encoder to extract audio features. These features capture information such as the rhythm, tone, and pronunciation of the speech, which will be used to drive the facial expressions and head movements of the character in the video.
4. Diffusion Process: During the diffusion process, the main network receives multiple frames of noise as input and attempts to denoise these noises into continuous video frames at each time step. This process involves two main attention mechanisms: Reference-Attention and Audio-Attention. Reference-Attention is used to maintain character identity consistency, while Audio-Attention is used to modulate the character's movements.
5. Temporal Module: To handle the temporal dimension and adjust the speed of motion, EMO uses temporal modules. These modules operate on features within frames through a self-attention mechanism to capture the dynamic content of the video and ensure coherence and consistency between consecutive frames.
6. Face Localization and Speed Control: To ensure the stability and controllability of the generated character motions, EMO uses a Face Locator and Speed Layers. The Face Locator encodes the facial bounding box region through lightweight convolutional layers, while the Speed Layers control the speed and frequency of motion by embedding head rotation speed into the generation process.
7. Training Strategy: The training of EMO is divided into three stages: image pretraining, video training, and speed layer integration. In the image pretraining stage, the main network and ReferenceNet learn from single-frame images. In the video training stage, temporal modules and audio layers are introduced to process consecutive frames. Finally, in the speed layer integration stage, only the temporal modules and speed layers are trained to ensure the ability of audio to drive character motions.
8. Generate Video: In the inference stage, EMO uses the DDIM sampling algorithm to generate video clips. Through an iterative denoising process, it ultimately generates a portrait video synchronized with the input audio.