What is ACTalker
ACTalker is an end-to-end video diffusion framework for generating realistic talking head videos. It supports single-signal or multi-signal control, such as audio, expression, etc. The core architecture includes a parallel Mamba structure, which uses multiple branches to control different facial regions with different driving signals, and is based on a gating mechanism and mask-drop strategy to achieve flexible and natural video generation. On the CelebV-HQ dataset, ACTalker achieves excellent Sync-C and Sync-D scores of 5.317 and 7.869, respectively, and an FVD-Inc score of 232.374, demonstrating good audio synchronization and video quality.
ACTalker's Main Features
- Multi-Signal Control and Single-Signal Control: ACTalker supports multi-signal control and single-signal control, and can use audio, expressions, and other signals to drive the generation of talking head videos.
- Natural and Coordinated Video Generation: Through the Mamba structure, driving signals can manipulate feature tokens across both temporal and spatial dimensions in each branch, ensuring the natural temporal and spatial coordination of the controlled video.
- High-Quality Video Generation: Experimental results show that ACTalker can generate natural and realistic facial videos. Under multi-signal control, the Mamba layers can seamlessly integrate multiple driving modalities and generate videos without conflict.
ACTalker's Technical Principles
- Parallel Mamba Structure: ACTalker adopts a parallel Mamba structure containing multiple branches, each of which uses a separate driving signal (such as audio, expression, etc.) to control a specific facial region. This allows signals from different modalities to act simultaneously on the video generation process without interfering with each other, enabling multi-signal control.
- Gating Mechanism: A gating mechanism is applied in all branches, randomly enabled or disabled during training and manually adjustable as needed during inference. The gating mechanism provides a flexible way to control video generation, supporting the choice of using a single signal or multiple signals for driving in different situations.
- Mask-Drop Strategy: ACTalker introduces a mask-drop strategy, which supports each driving signal independently controlling its corresponding facial region. During training, the strategy randomly drops feature tokens unrelated to the controlled region, enhancing the effectiveness of the driving signals, improving the quality of the generated content, and preventing control conflicts.
- State Space Modeling (SSM): To ensure the natural temporal and spatial coordination of the controlled video, ACTalker adopts state space modeling (SSM). The model supports driving signals manipulating feature tokens across both temporal and spatial dimensions in each branch, achieving natural coordination of facial movements.
- Video Diffusion Model Foundation: ACTalker is built on a video diffusion model and introduces a multi-branch control module in the denoising process. Each Mamba branch processes a specific modality signal and dynamically adjusts the influence weight of each modality through the gating mechanism.
ACTalker's Project Address
- Project Website：https://harlanhong.github.io/publications/actalker
- Github Repository：https://github.com/harlanhong/ACTalker
- HuggingFace Model Library：https://huggingface.co/papers/2504.02542
- arXiv Technical Paper：https://arxiv.org/pdf/2504.02542
ACTalker's Application Scenarios
- Virtual Anchors: ACTalker can generate natural and smooth talking head videos controlled by multiple signals such as audio and facial expressions, making virtual anchors more vivid and realistic, better interacting with the audience, and improving the audience's viewing experience.
- Remote Meetings: In remote meetings, ACTalker can generate natural talking-head videos using audio signals and participants' facial expression signals. It can solve the problem of lip and voice desynchronization caused by network latency, enabling participants to generate natural facial videos from audio and expression signals when the video signal is poor, enhancing the sense of realism in remote communication.
- Online Education: In online education scenarios, teachers can use ACTalker to generate natural talking-head videos. Through the control of audio and expression signals, teaching videos can be made more vivid and interesting, attracting students' attention and improving teaching effectiveness.
- Virtual Reality and Augmented Reality: In virtual reality (VR) and augmented reality (AR) applications, ACTalker can generate talking-head videos that match the virtual environment or augmented reality scene.
- Entertainment and Gaming: In the fields of entertainment and gaming, ACTalker can generate natural talking-head videos for characters, enhancing the characters' expressiveness and sense of immersion.