AudioX-Turbo - An audio generation framework jointly launched by Noiz AI and Tsinghua University

What is AudioX-Turbo

AudioX-Turbo is a unified and efficient audio generation framework jointly launched by Noiz AI, The Hong Kong University of Science and Technology, and Tsinghua University. It supports generating high-quality sound effects and music from any combination of text, video, and audio inputs. The model is based on a 2.7B-parameter multimodal diffusion Transformer architecture. Through distribution matching distillation and adversarial distillation, it compresses the inference steps from 50–200 steps to 4 steps, generating 10 seconds of audio in only 0.24 seconds on a single RTX 4090. The team built IF-caps-Pro, a strongly instruction-following dataset of about 9.2 million samples, achieving precise timestamp control for the first time.

Main features of AudioX-Turbo

• Text-to-Audio (T2A): Generates ambient sounds, action sound effects, etc. from plain text prompts, supporting precise description of sound type, style, and scene.
• Text-to-Music (T2M): Generates music clips with specified style, instrument configuration, rhythm, and mood based on text instructions, supporting complex music theory descriptions.
• Video-to-Audio (V2A): Automatically dubs and creates Foley for silent videos, matching action sound effects and ambient sounds in real time according to the visual content.
• Video-to-Music (V2M): Analyzes video emotion, rhythm, and visual dynamics, automatically generating synchronized background music and supporting multiple music styles.
• Joint Text+Video Generation (TV2A / TV2M): Combines video visuals and text instructions for precise control, for example specifying “thunder at the 3rd second, add guitar at the 5th second,” enabling timestamp-level precise sound effect/music generation.
• Audio Completion and Restoration : Performs completion, restoration, or style transfer based on contextual audio clips, supporting intelligent continuation and denoising of audio content.
• Image-to-Audio (Zero-Shot): Requires no dedicated training for images and can infer and generate corresponding ambient sounds or scene sound effects from static image content.

Follow on WeChat and reply “Open Source” to join theAI Open Source Project Exchange Group

Technical principles of AudioX-Turbo

• Multimodal Diffusion Transformer Architecture: The model uses a 2.7B-parameter multimodal diffusion Transformer built on the Flow Matching framework. Text semantic features are extracted through the T5 encoder, video temporal visual representations are extracted via Synchformer, and audio is compressed by a VAE into latent representations. After unified projection, the three modalities are fed into a shared Transformer for cross-attention fusion, enabling joint generation under any combination of text, video, and audio conditions.
• Distribution Matching Distillation and Adversarial Distillation: To compress the teacher model’s 50–200-step sampling process to 4 steps, the team adopted a two-stage distillation strategy: first, distribution matching distillation (DMD) trains the student model to approximate the teacher model’s flow field distribution, so that a single-step prediction can match multi-step results; then, a diffusion discriminator is introduced for adversarial distillation to further refine audio details under the 4-step constraint. Some objective metrics of the student model even surpass those of the teacher model.
• Strong Instruction Dataset IF-caps-Pro: The team built the IF-caps-Pro dataset of approximately 9.2 million samples, with the core innovation lying in structured annotation: each data entry comes with metadata such as precise timestamps, event counts, instrument types, and order of appearance. This enables the model not only to understand static combinations like "guitar + drum beats," but also to precisely execute complex temporal instructions such as "cicada chirps for 3 seconds first, guitar joins at the 5th second, fades out at the 8th second," significantly improving controllable generation capability.
• Three-stage progressive training strategy: Training is divided into three stages: in the first stage, pretraining is performed on pure text-audio data to establish basic acoustic understanding; in the second stage, video data is introduced to extend to multimodal joint generation and learn audio-visual synchronization relationships; in the third stage, distillation acceleration is performed to compress the multi-step model into a 4-step ultra-fast version. This progressive strategy ensures that the model does not lose multimodal understanding capability while compressing inference cost.

How to use AudioX-Turbo

• Environment setup: Clone the GitHub repository and create a Python 3.8 environment, install FFmpeg, libsndfile, and the dependency packages in requirements.txt.
• Weight download: Download the pretrained weights of the AudioX-Turbo 4-step student model, VAE, and Synchformer video encoder from Hugging Face or GitHub Release.
• Launch inference: Run python run_gradio.py to launch the local Web interface, or directly call the Python API to load the model and tokenizer.
• Parameter settings: Set video_path、text_prompt、audio_path and other input parameters according to the task type; the model supports arbitrary combinations of text, video, and audio inputs.
• Generate audio: Execute forward inference; 4-step sampling can generate 10 seconds of high-quality audio, taking about 0.24 seconds on an RTX 4090.

Core advantages of AudioX-Turbo

• Ultra-fast inference: 4-step sampling can match the audio quality of the teacher model's 100 steps, reducing NFE by about 25 times and achieving near-real-time generation.
• Unified multimodality: A single model supports arbitrary combinations of text, video, and audio inputs, with no need to separately train expert models for different tasks.
• Precise instruction following: 9.2 million structured data entries with timestamps, event counts, and instrument annotations enable the model to precisely understand complex temporal instructions such as "cicada chirps first, then guitar."
• High-quality distillation: Flow Matching-based Distribution Matching Distillation (DMD) combined with a diffusion discriminator; the student model surpasses the teacher model on some metrics.

AudioX-Turbo project address

• GitHub repository：https://github.com/NoizAI/AudioX-Turbo
• HuggingFace model library：https://huggingface.co/HKUSTAudio/AudioX-Turbo
• arXiv technical paper：https://arxiv.org/pdf/2606.12555