What is MEMO
MEMO (Memory-Guided EMOtion-aware diffusion) is an audio-driven portrait animation framework launched by Skywork AI, Nanyang Technological University, and the National University of Singapore, used to generate talking videos with identity consistency and expressiveness. MEMO is built around two core modules: a memory-guided temporal module and an emotion-aware audio module. The memory-guided module enhances identity consistency and motion smoothness by storing longer-term motion information, while the emotion-aware module uses a multimodal attention mechanism to improve the interaction between audio and video, refining facial expressions based on the emotion in the audio. Across talking videos with various image and audio types, MEMO demonstrates superior overall quality, audio-lip synchronization, identity consistency, and expression-emotion alignment compared to existing state-of-the-art methods.
Main Features of MEMO
- Audio-driven portrait animation: MEMO generates synchronized talking videos with identity consistency based on the input audio and reference image.
- Diverse content generation: Supports talking video generation for multiple image styles (such as portraits, sculptures, digital art) and audio types (such as speech, singing, rap).
- Multilingual support: Can process audio input in multiple languages, including English, Mandarin, Spanish, Japanese, Korean, and Cantonese.
- Expression-rich video generation: Generates talking videos with corresponding expressions based on the emotional content of the audio.
- Long video generation capability: Can generate long-duration talking videos with little error accumulation.
Technical Principles of MEMO
- Memory-guided temporal module：
- Memory state: Develops a memory state to store information from longer past context, guiding temporal modeling.
- Linear attention: Uses long-term motion information based on a linear attention mechanism, improving the coherence of facial motion and reducing error accumulation.
- Emotion-aware audio module：
- Multimodal attention: Simultaneously processes video and audio input, enhancing the interaction between the two.
- Audio emotion detection: Dynamically detects emotional cues in the audio, integrating emotional information into the video generation process to refine facial expressions.
- End-to-end framework：
- Reference Net: Provides identity information, used in spatial and temporal modeling.
- Diffusion Net: Where the core innovation lies, containing the memory-guided temporal module and the emotion-aware audio module.
- Data processing pipeline: Includes steps such as scene transition detection, face detection, image quality assessment, and audio-lip synchronization detection, ensuring data quality.
- Training strategy: Divided into two stages: facial domain adaptation and emotion-decoupled robust training, trained using rectified flow loss.
MEMO Project Address
- Project official website：memoavatar.github.io
- GitHub repository：https://github.com/memoavatar/memo
- HuggingFace model repository：https://huggingface.co/memoavatar/memo
- arXiv technical paper：https://arxiv.org/pdf/2412.04448
Application Scenarios of MEMO
- Virtual assistants and chatbots: Generate realistic videos of virtual assistants or chatbots, making them more natural and friendly when communicating with users.
- Entertainment and social media: In the entertainment industry, create dynamic video content for virtual idols, game characters, or social media influencers.
- Education and training: Generate educational videos in which the image of the teacher or trainer changes dynamically according to the teaching content, improving the interactivity and appeal of the learning experience.
- News and media: In news broadcasting, generate videos of the anchor, especially when multilingual broadcasting is required, quickly generate anchor videos in the corresponding language.
- Advertising and marketing: Create customized advertising videos, with product spokespersons personalized and adjusted according to different markets and audience groups.