What is Hallo3
Hallo3 is a portrait image animation technology jointly launched by Fudan University and Baidu, based on Diffusion Transformer Networks, capable of generating highly dynamic and realistic videos. Hallo3 is built on a pretrained transformer video generation model and effectively addresses the challenges of existing methods in handling non-frontal viewpoints, dynamic object rendering, and generating immersive backgrounds. Based on a new video backbone network, Hallo3 designs an identity reference network to ensure facial identity consistency throughout the video sequence, and studies speech audio conditions and motion frame mechanisms to achieve continuous video generation driven by speech audio. Experiments show that Hallo3 performs excellently in generating realistic portraits with multiple orientations, can adapt to complex poses and dynamic scenes, and produces realistic and smooth animation effects.
Main features of Hallo3
- Multi-view animation generation: Generate animated portraits from different viewpoints (such as frontal, side, overhead, or low angle), breaking through the limitation of traditional methods that mainly rely on frontal viewpoints.
- Dynamic object rendering: Handle the animation of dynamic objects around the portrait, such as a person holding a smartphone, a microphone, or wearing tightly fitting items, and generate the real motion of the objects in the video sequence.
- Immersive background generation: Generate backgrounds with dynamic effects, such as a campfire in the foreground or a crowded street scene in the background, enhancing the realism and immersion of the video.
- Identity consistency preservation: Maintain the identity consistency of the portrait throughout the entire video sequence, and accurately recognize and preserve the person's facial features even in long-duration animations.
- Speech-driven animation: Drive the portrait's facial expressions and lip movements based on speech audio, achieving highly synchronized speech and facial animation, making the animation more natural and realistic.
Technical principles of Hallo3
- Pretrained transformer video generation model：
- Base architecture: The CogVideoX model serves as the base architecture, using a 3D variational autoencoder (VAE) to compress video data. The model combines latent variables with text embeddings and processes them based on an expert transformer network.
- Conditioning mechanism: Introduces three conditioning mechanisms: text prompt (ctext), speech audio condition (caudio), and identity appearance condition (cid). It mainly uses cross-attention and adaptive layer normalization (adaLN) to integrate these conditioning information.
- Identity reference network：
- 3D VAE and transformer layers: Uses an identity reference network combining a causal 3D VAE with 42 transformer layers to extract identity features from the reference image, embed them into the denoising latent code, and enhance the model's representation and long-term preservation of identity information based on the self-attention mechanism.
- Feature fusion: Fuses the visual features generated by the reference network with the features of the denoising network to ensure that the generated facial animation maintains consistency and coherence over long sequences.
- Speech audio conditions：
- Audio embedding: The wav2vec framework extracts audio features and generates frame-specific audio embeddings, and a linear transformation layer converts the audio embeddings into a representation suitable for the model.
- Cross-attention mechanism: In the denoising network, the cross-attention mechanism interacts audio embeddings with latent codes, enhancing the coherence and relevance of the generated output and ensuring that the model effectively captures the audio signals driving character generation.
- Video Extrapolation: Introduce motion frames as conditioning information. The last few frames of the generated video are used as input for generating subsequent segments, and the motion frames are processed with a 3D VAE to generate new latent codes, enabling temporally consistent long-video inference.
- Training and Inference：
- Training Process: It is divided into two stages. In the first stage, the model is trained to generate videos with identity consistency; in the second stage, it is extended to audio-driven video generation, integrating an audio attention module.
- Inference Process: The model takes a reference image, driving audio, text prompt, and motion frames as input to generate videos with identity consistency and lip synchronization.
Hallo3 Project Address
- Project Official Website：https://fudan-generative-vision.github.io/hallo3
- GitHub Repository：https://github.com/fudan-generative-vision/hallo3
- HuggingFace Model Library：https://huggingface.co/fudan-generative-ai/hallo3
- arXiv Technical Paper：https://arxiv.org/pdf/2412.00733
Hallo3 Application Scenarios
- Game Development: Generate dynamic portrait animations for characters in games, making their performance in the game more natural and realistic and enhancing the player's gaming experience.
- Film Production: Generate realistic character animations to enhance the visual effects and immersion of films and animations.
- Social Media: Generate dynamic avatars for social media users, making their profiles more vivid and interesting and enhancing their personalized experience on social media.
- Online Education: Generate animations of virtual instructors to make online courses more vivid and interesting and increase students' interest and participation in learning.
- Virtual Reality and Augmented Reality: Generate virtual characters in VR and AR applications to provide more realistic interactive experiences and enhance users' sense of immersion and participation.