What is AudioGen-Omni
AudioGen-Omni is a multimodal audio generation framework launched by Kuaishou. The framework can generate high-quality audio, speech, and songs based on inputs such as video and text. Through a unified lyrics-text encoder and Phase-Aligned Anisotropic Positional Injection (PAAPI) technology, the framework achieves precise audiovisual alignment and cross-modal synchronization. The framework supports multilingual input, has fast inference speed, can generate 8 seconds of audio in 1.91 seconds, and performs excellently on a variety of audio generation tasks, making it suitable for scenarios such as video dubbing, speech synthesis, and song creation.
Main Features of AudioGen-Omni
- Multimodal audio generation: Generates high-quality audio, speech, and songs based on video, text, or a combination of both.
- Precise audiovisual alignment: Based on Phase-Aligned Anisotropic Positional Injection (PAAPI) technology, it achieves lip-sync and rhythm alignment between audio and video.
- Multilingual support: Supports input in multiple languages and generates speech and songs in the corresponding language.
- Efficient inference: Fast inference speed, generating 8 seconds of audio in 1.91 seconds, significantly outperforming similar models.
- Flexible input conditions: Can handle cases of missing modalities; even with only video or only text input, it can generate stable audio output.
- High-quality audio generation: The generated audio matches the input highly in semantics and acoustic performance, supporting high-fidelity audio generation.
Technical Principles of AudioGen-Omni
- Multimodal Diffusion Transformer (MMDiT): Integrates video, audio, and text modalities into a shared semantic space, supporting a variety of audio generation tasks. Based on a joint training paradigm, it uses large-scale video-text-audio data to enhance cross-modal associations.
- Unified lyrics-text encoder: Encodes graphemes and phonemes into frame-level dense representations, adapting to speech and singing tasks. It uses multilingual unified tokenization and ConvNeXt refinement to generate frame-aligned representations.
- Phase-Aligned Anisotropic Positional Injection (PAAPI): Selectively applies Rotary Position Embedding (RoPE) to temporal modalities (such as video and audio), improving the precision of cross-modal temporal alignment.
- Dynamic conditioning mechanism: Based on unfreezing all modalities and masking missing inputs, it avoids the semantic limitations of the text-frozen paradigm and supports flexible multimodal conditional generation.
- Joint attention mechanism: Based on AdaLN (Adaptive Layer Normalization), it enhances cross-modal feature fusion and promotes cross-modal information exchange through a joint attention mechanism.
Project Address of AudioGen-Omni
- Project official website：https://ciyou2.github.io/AudioGen-Omni/
- arXiv technical paper：https://arxiv.org/pdf/2508.00733
Application Scenarios of AudioGen-Omni
- Video dubbing: Automatically generates precisely matching speech, songs, or sound effects for videos, improving video creation efficiency and content richness.
- Speech synthesis: Quickly converts text into natural and fluent speech, suitable for fields such as audiobooks, voice assistants, and intelligent customer service.
- Song creation: Generates matching songs based on video content or lyrics, assisting music creation and enriching video background music.
- Sound effect generation: Generates natural environmental sound effects, action sound effects, etc., based on text descriptions or video content, enhancing the immersive feel of the content.