What is VASA-1
VASA-1 is a generation framework proposed by Microsoft Research Asia that converts static photos intolip-syncdynamic videos, capable of generating realistic 3D talking facial animations in real time based on a single static face photo and a segment of speech audio. Through precise lip-speech synchronization, rich facial expression details, and natural head movements, this technology creates highly realistic and lively virtual characters. The core innovation of VASA-1 lies in its holistic facial dynamics and head motion generation model, which operates in a facial latent space, efficiently generating high-resolution videos while supporting online generation and low latency.
VASA-1 Features
- Realistic lip-sync: VASA-1 can generate lip movements precisely synchronized with the input speech audio, providing a highly realistic talking effect.
- Rich facial expressions: In addition to lip movements, VASA-1 can also capture and reproduce a series of complex facial expressions and subtle emotional details, increasing the realism of the animation.
- Natural head movements: The model can simulate natural head movements, such as turning and tilting, making the generated talking face videos more vivid.
- Efficient video generation: VASA-1 supports online generation of 512×512 resolution videos at up to 40 FPS, with extremely low initial latency, suitable for real-time applications.
- Flexible and controllable generation: By accepting optional signals as conditions, such as main gaze direction, head distance, and emotion offset, VASA-1 can control the generation process, improving the diversity and adaptability of the output.
- Handling different inputs: VASA-1 can handle photo and audio inputs outside the training distribution, such as artistic photos, singing audio, and non-English speech.
VASA-1 Official Website Entry
- Official Project Homepage：https://www.microsoft.com/en-us/research/project/vasa-1/
- arXiv Research Paper：https://arxiv.org/abs/2404.10667
How VASA-1 Works
- Input Preparation: VASA-1 accepts two main inputs: a static facial image of any individual and a speech audio clip from any person.
- Facial Feature Extraction: Uses a face encoder to extract features such as 3D appearance volume, identity code, head pose, and facial dynamics code from the input static facial image.
- Facial Latent Space Modeling: Constructs a facial latent space that is highly disentangled between facial dynamics and other factors (such as identity and appearance), and has the expressive capacity for rich expression details and dynamic nuances.
- Diffusion Model Training: Trains a diffusion-based model (Diffusion Transformer) that can generate comprehensive facial dynamics and head movements in the facial latent space, conditioned on the given audio and optional control signals.
- Conditional Signal Integration: Feeds control signals such as main gaze direction, head distance, and emotion offset as conditions into the diffusion model to guide the generation of facial dynamics.
- Facial Dynamics and Head Movement Generation: Utilizes the trained diffusion model to generate latent code sequences of facial dynamics and head movements based on the input audio features and conditional signals.
- Video Frame Generation:Using the face decoder and the appearance and identity features extracted from the encoder, produce the final video frames based on the generated facial dynamics and head motion latent codes.