MOFA-Video - Tencent's Open-Source Controllable AI Image-to-Video Model

What is MOFA-Video

MOFA-Video is a controllable image-to-video generation model open-sourced by researchers from Tencent AI Lab and the University of Tokyo. This technology uses a generative motion field adapter to animate images and generate videos. Building on the pretrained Stable Video Diffusion model, MOFA-Video can achieve fine-grained control over motion during video generation through sparse control signals such as manual trajectories, facial landmark sequences, or audio. MOFA-Video can not only use these control signals individually but also combine them to create more complex animations in a zero-shot manner, providing a new, highly controllable image animation video solution.

MOFA-Video's Features

• Trajectory-Controlled Animation: Users guide MOFA-Video to generate corresponding video animations by manually drawing trajectories on images. This feature is particularly suitable for scenarios requiring precise control of object or camera movement.
• Facial Landmark Animation: The system uses facial landmark data, such as markers obtained through facial recognition technology, to generate realistic facial expressions and head movement animations.
• Hybrid Control Animation: MOFA-Video can combine trajectory control and facial landmark control to achieve synchronized animation of facial expressions and body movements, creating complex multi-part animation effects.
• Audio-Driven Facial Animation: By analyzing audio signals, MOFA-Video can generate facial animations synchronized with speech or music, such as lip-syncing.
• Video-Driven Facial Animation: Using a reference video, MOFA-Video can make facial movements in a static image mimic the movements in the video, achieving dynamic facial expression reproduction.
• Zero-Shot Multimodal Control: MOFA-Video supports zero-shot learning, meaning different control signals can be combined without additional training, greatly improving the flexibility and diversity of animation generation.
• Long Video Generation Capability: By adopting a periodic sampling strategy, MOFA-Video can generate longer video animations than traditional models, breaking through frame count limitations.
• User Interface Operation: MOFA-Video provides a simple and easy-to-use Gradio-based user interface, through which users can intuitively generate animations without professional programming skills.

MOFA-Video's Official Website Entry

• Official Project Homepage:https://myniuuu.github.io/MOFA_Video
• GitHub Code Repository:https://github.com/MyNiuuu/MOFA-Video
• Trajectory-Based Image Animation Gradio Demo and Model Checkpoints:https://huggingface.co/MyNiuuu/MOFA-Video-Traj
• Gradio Demo and Hybrid Control Image Animation Checkpoints:https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid

How MOFA-Video Works

1. Sparse Control Signal Generation: During the training phase, the system generates sparse control signals through sparse motion sampling techniques. These signals may be trajectory-based animation control points, facial landmark sequences, or other forms of motion instructions.
2. MOFA-Adapter Design: MOFA-Adapter is the core of the system. It is a specially designed network structure used to convert sparse control signals into dense motion fields. This component includes:
   • S2D Network: Converts sparse motion prompts into dense motion fields.
   • Reference Image Encoder: Extract multi-scale features from the reference image for subsequent motion field generation.
   • Feature Fusion Encoder: Combine the motion field generated by the S2D network with the features of the reference image encoder.
3. Multi-Scale Feature Extraction: The reference image encoder processes the input reference image and extracts multi-scale feature representations, which will be used for guidance and deformation in the subsequent video frame generation process.
4. Generation and Application of Motion Fields: The S2D network generates dense motion fields based on sparse control signals, and these motion fields are then used to spatially deform multi-scale features to simulate motion effects in the video.
5. Pretrained SVD Model: MOFA-Adapter is combined with the pretrained Stable Video Diffusion model (SVD), using the conditional features obtained from MOFA-Adapter to guide the generation of video frames.
6. Spatial Deformation: Using the generated motion fields, the system spatially deforms the multi-scale features of the reference image to ensure that objects and scene elements in the video frames move according to the predetermined motion trajectories.
7. Video Frame Generation: The features deformed in the feature space are used to generate video frames. This process involves sampling from the latent space and gradually removing noise to reconstruct clear video frames.
8. Multimodal Control Signal Integration: MOFA-Video can process control signals from different sources and fuse them into a unified generation process to achieve complex animation effects.
9. Zero-Shot Learning: After MOFA-Adapter training is completed, it can work jointly across different control domains without additional training, achieving fine-grained control over video generation.
10. Long Video Generation Strategy: To generate longer videos, MOFA-Video adopts a periodic sampling strategy, which addresses the issues of coherence and computational complexity in long video generation by grouping and overlapping sampling of frames in the latent space.