daVinci-MagiHuman - An open-source audio-video generation model from Sand.ai and others

What is daVinci-MagiHuman?

daVinci-MagiHuman is an open-source joint audio-video generation foundation model jointly released by the GAIR Lab at Shanghai Innovation Institute and Sand.ai. The model adopts a 15-billion-parameter single-stream Transformer architecture that uniformly models three modalities—text, video, and audio—without requiring cross-attention mechanisms. The model excels at character-centric generation and supports multiple languages including Chinese, English, Japanese, Korean, German, and French. It can generate a 5-second 256p video in 2 seconds on a single H100. Compared with Ovi 1.1 and LTX 2.3, it achieves win rates of 80% and 60.9%, respectively. The code, model weights, and online demo have all been fully open-sourced.

Main Features of daVinci-MagiHuman

• Joint audio-video generation: Supports synchronized generation of character videos with natural speech and lip synchronization, achieving true integrated audio-visual output.
• Multilingual support: Supports speech generation in multiple languages including Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French.
• Portrait performance generation: Focuses on character-centric scenes, generating expressive facial expressions, body movements, and emotional conveyance.
• Ultra-fast inference: Supports generating a 5-second 256p video in 2 seconds on a single H100 GPU, meeting real-time interaction needs.
• High-resolution output: Through latent-space super-resolution technology, it can be extended to 540p or 1080p HD video.

Technical Principles of daVinci-MagiHuman

• Single-stream unified architecture: daVinci-MagiHuman adopts a single-stream Transformer architecture that places text, video, and audio uniformly into the same 15-billion-parameter, 40-layer denoising network, using pure self-attention mechanisms for joint modeling and completely abandoning cross-attention or modality-specific branches. Architecturally, it uses a "sandwich" design: a few layers at the beginning and end retain modality-related parameters, while the middle backbone network shares parameters, achieving a balance between modality specialization and deep fusion; it also introduces mechanisms such as injection without explicit timestep conditioning and Attention-Head gating to improve training stability and expressive capability.
• Latent-space super-resolution: The model uses a two-stage pipeline: the base model first generates low-resolution audio-video latents, and then latent-space super-resolution directly performs high-resolution refinement in latent space, avoiding additional VAE encoding and decoding overhead. The audio latents continue to be fed as input into the super-resolution model to maintain lip-sync performance.
• Inference acceleration optimization: During inference, a lightweight Turbo VAE decoder is used to reduce latency, and the self-developed MagiCompiler is integrated for full-graph compilation optimization, bringing about 1.2x acceleration through cross-layer operator fusion; combined with DMD-2 distillation technology, it achieves high-quality generation with only 8 denoising steps.

Key Information and Usage Requirements of daVinci-MagiHuman

• Model scale: 15 billion parameters, 40-layer Transformer
• Architecture features: Single-stream unified architecture, pure self-attention, no cross-attention
• Generation capabilities: Supports text/image-driven joint generation of portrait audio and video
• Supported languages: Chinese (Mandarin, Cantonese), English, Japanese, Korean, German, French
• Inference speed: Generates a 5-second 256p video in 2 seconds on a single H100, and a 1080p video in 38 seconds
• Performance: 80.0% win rate against Ovi 1.1, 60.9% win rate against LTX 2.3
• Hardware: NVIDIA GPU (H100 recommended), CUDA support required
• Software Environment：Python 3.12，PyTorch 2.9.0，CUDA 12.x
• Dependencies: Flash Attention (Hopper architecture), MagiCompiler (in-house compiler), Turbo VAE