What is MirrorMe
MirrorMe is a real-time, high-fidelity audio-driven portrait animation framework launched by the Alibaba Tongyi Lab. The framework is based on the LTX video model and combines three key innovations—identity injection mechanism, audio-driven control module, and progressive training strategy—to address the challenge of real-time generation of high-fidelity, temporally consistent animated videos. MirrorMe topped the EMTD benchmark, demonstrating excellent image fidelity, lip-sync accuracy, and temporal stability, with efficient inference speed that meets real-time generation needs, providing strong technical support for application scenarios such as e-commerce livestreaming.
Main Features of MirrorMe
- Real-Time High-Fidelity Animation Generation: MirrorMe can generate high-quality upper-body animation videos in real time, supporting smooth playback at 24FPS to meet the needs of real-time interaction.
- Audio-Driven Lip Sync: MirrorMe can accurately convert audio signals into corresponding mouth movements, achieving highly realistic lip sync.
- Identity Preservation: Based on the reference identity injection mechanism, MirrorMe ensures that the generated animation video is highly consistent in appearance with the input reference image.
- Precise Expression and Gesture Control: MirrorMe can control facial expressions and, based on gesture signals, achieve precise control of hand movements.
Technical Principles of MirrorMe
- Base Architecture: Based on the LTX video model as the core architecture. The LTX model is a video generation model based on a Diffusion Transformer, using spatiotemporal tokenization (each token corresponds to 32×32×8 pixels) to achieve an extremely high compression ratio (1:8192).
- Identity Injection Mechanism: To ensure that the generated video is highly consistent in appearance with the input reference image, MirrorMe is based on a reference identity injection mechanism. The reference image is encoded using the 3D Variational Autoencoder (VAE) in the LTX model to obtain reference latent variables. The encoded reference latent variables are concatenated with the noisy latent variables along the temporal dimension, and identity information is injected based on the self-attention mechanism. This ensures that the generated video maintains consistency in appearance with the reference image.
- Audio-Driven Control Module: Based on the audio-driven control module, precise synchronization between audio signals and video generation is achieved. The specific implementation is as follows:
- Causal Audio Encoder: A pretrained wav2vec2 model is used to extract frame-level audio embeddings from the raw audio signal, and the causal audio encoder progressively compresses the audio sequence so that its temporal resolution matches the video latent variables.
- Audio Adapter: The extracted audio features are injected into the video generation process through the audio adapter. The audio adapter fuses audio features with video features based on the cross-attention mechanism, ensuring that the audio signal can precisely drive facial expressions and mouth movements.
- Progressive Training Strategy: To improve the model's training effectiveness and generation quality, MirrorMe is based on a progressive training strategy. First, training is conducted on facial close-up images, focusing on learning the mapping relationship between audio and facial expressions. On the basis of facial close-up training, it gradually expands to upper-body synthesis, while introducing facial masks to maintain the dynamic response of the facial region. Based on the pose encoder module, hand keypoint signals are introduced to achieve precise control of hand movements.
- Efficient Inference: MirrorMe is based on the LTX model's efficient compression and denoising technology, significantly improving inference speed. Temporally, it compresses the input video to 1/8 of its original length, and spatially, it compresses it in both height and width to 1/32 of the original resolution, greatly reducing the number of latent variables that need to be processed. On consumer-grade NVIDIA GPUs, MirrorMe can achieve real-time generation at 24FPS, meeting the strict latency requirements of real-time applications.
MirrorMe's project address
- arXiv technical paper：https://arxiv.org/pdf/2506.22065v1
MirrorMe's application scenarios
- E-commerce livestreaming: Generate realistic virtual streamers, generating natural expressions and movements in real time based on input audio, enhancing livestream interactivity and appeal.
- Virtual customer service: In online customer service scenarios, the generated virtual customer service avatars interact with users in real time based on audio-driven input, providing a natural and friendly service experience, and supporting multiple languages, enabling them to serve users from different language backgrounds.
- Online education: Virtual teachers generated by MirrorMe generate expressions and movements in real time based on the teaching content, making teaching more vivid and interesting. Generating virtual avatars similar to students for personalized learning provides teaching content close to students' needs.
- Virtual meetings: MirrorMe generates virtual participant avatars, generating expressions and movements in real time based on participants' speech, enhancing meeting interactivity and engagement. It is suitable for remote collaboration and enhances the sense of face-to-face communication among team members.
- Social media: Users generate their own virtual avatars, interact on social media through audio-driven input, share interesting video content, and enhance the fun and participation of livestreaming.