LongCat-Video-Avatar 1.5 - Meituan's Open-Source Digital Human Video Generation Model

What is LongCat-Video-Avatar 1.5

LongCat-Video-Avatar 1.5 is an audio-driven digital human video generation framework open-sourced by Meituan LongCat. The model is built on a 13.6B-parameter base video generation model LongCat-Video , upgrading the audio encoder from Wav2Vec2 to Whisper-Large-v3 for more precise lip synchronization; through Step Distillation technology, inference is compressed to 8 steps, supporting 480P/720P resolutions and INT8 quantization, delivering production-grade physical plausibility and temporal stability in scenarios such as single-person and multi-person interaction, singing performance, and stylized animation.

Main Features of LongCat-Video-Avatar 1.5

• Single-Audio-Driven Video Generation: Input a single audio track along with text or a reference image to generate highly dynamic, lip-sync-accurate talking-head videos.
• Multi-Audio Interactive Generation: Supports dual audio stream input, enabling natural interaction scenarios such as two-person dialogue and alternating speech.
• Video Continuation and Long Video Generation: Natively supports Video Continuation, enabling continuous generation of minute-level long videos based on existing clips without color drift.
• Stylization and Animation Adaptation: Generalizes to anime characters, animal figures, and complex real-world conditions, supporting 3D animation styles.
• Singing and Performance Generation: Supports music-driven dynamic expressions and stable full-body/half-body performances.
• Multi-Task Foundation Model: The underlying LongCat-Video uniformly supports text-to-video, image-to-video, and video continuation.

Technical Principles of LongCat-Video-Avatar 1.5

• Whisper-Large-v3 Audio Encoding: Replaces Wav2Vec2 with Whisper-Large-v3 to extract finer speech features, significantly improving lip-sync accuracy.
• Step Distillation for Fast Inference: Reduces inference steps to 8 through distilled sampling, greatly accelerating generation while ensuring quality.
• Coarse-to-Fine Spatiotemporal Generation Strategy: Employs Coarse-to-Fine generation along temporal and spatial axes, combined with Block Sparse Attention, achieving efficient 720p/30fps inference.
• Multi-Reward GRPO Reinforcement Learning: Based on Group Relative Policy Optimization and multi-reward model training, optimizing text alignment, visual quality, and motion coherence.
• INT8 Quantization and Context Parallelism: Supports INT8 quantization to reduce VRAM usage, compatible with single-GPU/multi-GPU context-parallel inference.

How to Use LongCat-Video-Avatar 1.5

• Environment Setup: Clone the repository and create a Conda environment, install PyTorch 2.6.0, FlashAttention-2, and project dependencies.
• Download the Model: Download via HuggingFace CLI LongCat-Video-Avatar-1.5 weights to the local ./weights directory.
• Single-Audio Generation: Prepare a JSON file containing the audio path, text prompt, and reference image, run run_demo_avatar_single_audio_to_video.py, specifying --model_type avatar-v1.5 --use_distill --use_int8 to enable the 1.5 model, distilled inference, and INT8 quantization.
• Multi-Audio Generation: Prepare a JSON file containing two audio tracks and corresponding character information, run run_demo_avatar_multi_audio_to_video.py, supporting both Merge (overlay) and Concatenation (splice) dual-audio modes.
• Video Continuation: Add in the single/multi-audio script --num_segments parameter, the model will automatically segment and continue, generating continuous videos with longer temporal sequences.
• WebUI Experience: Execute streamlit run ./run_streamlit.py to launch the visual interface for interactive generation and parameter adjustment.

Core Advantages of LongCat-Video-Avatar 1.5

• Leading Lip-Sync Precision: The Whisper-Large-v3 audio encoder delivers stronger mouth-shape accuracy and smoother facial expression transitions.
• Long-Video Identity Consistency: Maintains stable identity and full-body motion coherence during long talking-head shots and hand-object interaction scenes.
• Extremely Fast Inference: 8-step distilled inference significantly shortens generation time, suitable for business scenarios with higher real-time requirements.
• Open Source and Commercially Usable: Model weights and inference code are open-sourced under the MIT license, allowing free deployment and secondary development.
• VRAM-Friendly: INT8 quantization and multi-GPU parallel support lower the hardware barrier.

Project Addresses of LongCat-Video-Avatar 1.5

• Project Website：https://meigen-ai.github.io/LongCat-Video-Avatar-1.5-Page/
• GitHub Repository：https://github.com/meituan-longcat/LongCat-Video
• HuggingFace Model Repository：https://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5

Comparison of LongCat-Video-Avatar 1.5 with Similar Competitors

Application Scenarios of LongCat-Video-Avatar 1.5

• AI Voiceover and E-Commerce Marketing: Merchants can input product explanation audio and reference images to batch-generate digital human product-promotion videos with accurate lip-sync and natural expressions, greatly reducing the cost of featuring real people.
• Online Education and Virtual Instructors: Educational institutions can convert course audio into virtual instructor lecture videos with stable identity consistency, supporting long-duration explanations and gesture interactions, improving content reuse efficiency.
• Virtual Customer Service and Corporate Image: Enterprises can deploy dedicated digital human customer service, combined with multi-audio stream functionality to achieve visualized multi-turn conversations, for scenarios such as official website reception and intelligent outbound follow-up calls.
• Short Video and Social Media Creation: Creators input singing or performance audio to quickly generate highly dynamic, stylized (anime, realistic, etc.) virtual avatar short videos, suitable for platforms such as Douyin and Kuaishou.