What is Hallo2
Hallo2 is an audio-driven video generation model jointly launched by Fudan University, Baidu, and Nanjing University. It can combine a single reference image and an audio input lasting several minutes, and based on optional text prompts, adjust portrait expressions to generate high-resolution 4K videos synchronized with the audio. Hallo2 is based on advanced data augmentation techniques such as patch dropout and Gaussian noise to enhance the long-term visual consistency and temporal coherence of videos. Hallo2 implements vector quantization and temporal alignment techniques for latent codes to generate 4K resolution videos, and introduces semantic text labels as conditional input to improve the controllability and diversity of animation. Hallo2 conducts extensive experiments on multiple public datasets, demonstrating its ability to generate long-duration, high-resolution, rich, and controllable content.
Main Features of Hallo2
- Long-duration video generation: It can generate videos up to one hour long, addressing issues of appearance drift and temporal artifacts.
- High-resolution output: It achieves 4K resolution portrait video generation, providing clear visual details.
- Audio-driven animation: It drives portrait image animation based on audio input, achieving synchronization of lip movements and expressions.
- Text prompt adjustment: It introduces text prompts to adjust and refine portrait expressions, increasing the diversity and expressiveness of the animation.
- Data augmentation techniques: Based on patch dropout and Gaussian noise enhancement techniques, it improves the long-term visual consistency and temporal coherence of videos.
Technical Principles of Hallo2
- Patch-Drop Augmentation: By randomly discarding some image patches in the conditioning frame, it reduces the influence of the previous frame on the appearance of subsequent frames, maintaining visual consistency in long-duration video generation.
- Gaussian noise enhancement: On the basis of patch dropout, Gaussian noise is added to further increase the model's reliance on the appearance of the reference image, preserve motion information, and reduce accumulated artifacts and distortion.
- Vector Quantized Generative Adversarial Network (VQGAN): Based on vector-quantized latent codes and the application of temporal alignment techniques, Hallo2 can maintain coherence in the temporal dimension and generate high-quality 4K resolution videos.
- Semantic text labels: Hallo2 introduces adjustable semantic text labels as conditional input, enabling the model to generate specific expressions and actions according to text prompts, improving the controllability of the generated content.
- Cross-Attention Mechanism: The model can effectively integrate motion conditions, such as audio features and text embeddings, during the denoising process to generate images consistent with the conditional input.
Hallo2 Project Address
- Project website：fudan-generative-vision.github.io/hallo2
- GitHub repository：https://github.com/fudan-generative-vision/hallo2
- HuggingFace model library：https://huggingface.co/fudan-generative-ai/hallo2
- arXiv technical paper：https://arxiv.org/pdf/2410.07718v1
- Hallo3 portrait animation generation framework：https://ai-bot.cn/hallo3/
Application Scenarios of Hallo2
- Film and video production: In film production, Hallo2 generates or enhances characters' facial expressions and lip movements, and is used in science fiction and animated films that require a large number of virtual characters or special effects.
- Virtual assistants and digital humans: In fields such as customer service, education, and entertainment, Hallo2 can create realistic virtual assistants or digital humans, providing more natural and engaging interactive experiences.
- Game development: Game developers use Hallo2 to generate highly realistic character animations, improving the immersion of the game and the player's gaming experience.
- Social Media and Content Creation: Content creators use Hallo2 to create dynamic portrait videos for social media platforms, increasing the appeal and interactivity of the content.
- News and Broadcasting: Hallo2 can generate animated images of news anchors and, in situations requiring multilingual broadcasting, quickly generate lip movements and expressions for different languages.