SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

EMAGE - A Full-Body Co-Speech Gesture Generation Framework Jointly Launched by Tsinghua University and the University of Tokyo, Among Others

Author: AI小集 · Original source: AI工具集

What is EMAGE

EMAGE (Expressive Masked Audio-conditioned GEsture modeling) is a full-body co-speech gesture generation framework launched by Tsinghua University, the University of Tokyo, Keio University, and other institutions. EMAGE can generate complete full-body gestures synchronized with audio based on audio and partially masked gesture input, including facial expressions, body movements, hand gestures, and global motion. EMAGE is based on a masked audio gesture transformer to achieve joint training of audio-to-gesture generation and masked gesture reconstruction, effectively encoding audio and body gesture prompts. EMAGE uses four combined pretrained VQ-VAEs (Vector Quantized Variational AutoEncoders) to decode local facial and body movements, and decodes global translation based on a pretrained global motion predictor.

EMAGE

Main Features of EMAGE

  • Generate gestures synchronized with audio: Based on the input audio signal, generate full-body gestures that match the rhythm and semantics of the audio.
  • Accept partially predefined gesture input: Based on partial gestures provided by the user (such as specific actions or poses) as prompts, generate complete and natural gesture sequences.
  • Generate diverse gestures: Combine multiple pretrained VQ-VAEs (Vector Quantized Variational AutoEncoders) to generate rich and diverse gesture actions, avoiding monotonous and repetitive results.
  • Support full-body gesture generation: Support the generation of facial and upper-limb movements as well as lower-limb and global motion, achieving more natural and complete human animation effects.

Technical Principles of EMAGE

  • Masked audio gesture transformer: During training, part of the gesture data is randomly masked, and the model predicts the masked parts based on audio and other unmasked gesture information. The masking mechanism is similar to the BERT model in natural language processing and can enhance the model's understanding of local and global information. Training for audio-to-gesture generation and masked gesture reconstruction allows the model to better learn the mapping relationship between audio and gestures, while also using existing gesture information (even if partially masked) to generate more accurate gestures.
  • Audio feature fusion: Combine the rhythmic features of audio (such as pitch fluctuations, rhythm strength) and content features (such as word embeddings from text transcription), so that the generated gestures can both reflect the rhythm of the audio and be related to semantics. For example, when "walking" is mentioned, the model can generate corresponding walking gestures.
  • Pretrained VQ-VAE: Different parts of the body (such as the face, upper limbs, lower limbs) are modeled separately with independent VQ-VAEs, and each VQ-VAE is responsible for generating the movements of a specific part. This segmented modeling approach can better capture the unique relationship between different body parts and audio, improving the accuracy and diversity of generated gestures.
  • Discrete representation learning: Based on VQ-VAE, continuous gesture actions are converted into discrete representations, making it easier for the model to learn and generate. This discrete representation is similar to vocabulary in language, allowing the model to process and generate complex gesture sequences more efficiently.
  • Global motion predictor: In addition to the generation of local movements, a pretrained global motion predictor is used to estimate the global translation of the body (such as walking, turning, etc.), making the generated gestures natural in local movements and more coherent and realistic in overall motion.

EMAGE Project Address

Application Scenarios of EMAGE

  • Virtual Reality (VR) and Augmented Reality (AR): Generate natural gestures and facial expressions for virtual characters, enhancing the interactive experience in scenarios such as virtual socializing, virtual meetings, and virtual games, making virtual characters more vivid and realistic.
  • Animation Production: Assist in the production of animated films, TV series, and dynamic advertisements by automatically generating character gestures and facial expressions based on dialogue content, reducing animators' workload and improving production efficiency.
  • Virtual Customer Service and Digital Humans: Create natural and smooth gestures and facial expressions for online customer service, intelligent assistants, etc., making them more approachable and realistic when communicating with users, and improving the user experience.
  • Education and Training: In the education field, virtual teachers generate corresponding gestures based on the content being explained, helping students better understand knowledge; in training scenarios, generate realistic operational demonstration gestures to assist teaching.
  • Entertainment Industry: In game development, generate matching gestures and facial expressions based on character voice, enhancing the game's immersion and realism.