What is TANGOFLUX
TANGOFLUX is an efficient text-to-audio generation model jointly launched by the Singapore University of Technology and Design (SUTD) and NVIDIA. The model has approximately 515 million parameters and can generate up to 30 seconds of 44.1kHz audio in just 3.7 seconds on a single A40 GPU. TANGOFLUX uses the CLAP-Ranked Preference Optimization (CRPO) framework, based on iteratively generating and optimizing preference data to improve the model's audio alignment capability. The model demonstrates excellent performance in both objective and subjective benchmark tests, with code and models open-sourced on platforms such as GitHub to support further research.
Main Features of TANGOFLUX
- Efficient Audio Generation: TANGOFLUX can quickly generate high-quality audio content, generating up to 30 seconds of 44.1kHz audio within 3.7 seconds.
- Text-to-Audio Conversion: The model directly converts text descriptions into corresponding audio output, achieving direct text-to-audio conversion.
- Preference Optimization: TANGOFLUX can optimize audio output to better match user preferences and the intent of the input text.
- Training on Non-Proprietary Data: Trained on non-proprietary datasets, making the model more open and accessible.
Technical Principles of TANGOFLUX
- Variational Autoencoder: Uses a VAE to encode audio waveforms into latent representations, and reconstructs the original audio from the latent representations.
- Text and Duration Embedding: The model controls the content and duration of the generated audio based on text encoding and duration encoding, achieving controllable audio generation.
- FluxTransformer Architecture: Built on FluxTransformer blocks, combining Diffusion Transformer (DiT) and Multimodal Diffusion Transformer (MMDiT) to process text prompts and generate audio.
- Flow Matching (FM): Based on the flow matching framework, it learns the mapping from a simple prior distribution to a complex target distribution to generate samples.
- CLAP-Ranked Preference Optimization (CRPO): The CRPO framework iteratively generates preference data pairs to optimize audio alignment. It uses the CLAP model as a proxy reward model, evaluating the quality of audio output based on joint text and audio embeddings, and accordingly constructs a preference dataset for preference optimization.
- Direct Preference Optimization: TANGOFLUX applies DPO to flow matching, comparing winning and losing audio samples to optimize the model and improve the alignment between audio and text descriptions.
Project Addresses of TANGOFLUX
- Project Official Website：tangoflux.github.io
- GitHub Repository：https://github.com/declare-lab/TangoFlux
- HuggingFace Model Hub：https://huggingface.co/declare-lab/TangoFlux
- arXiv Technical Paper：https://export.arxiv.org/pdf/2412.21037
- Online Experience Demo：https://huggingface.co/spaces/declare-lab/TangoFlux
Application Scenarios of TANGOFLUX
- Multimedia Content Creation: In film, games, advertising, and video production, it is used to generate background music, sound effects, and voiceovers, improving production efficiency and reducing costs.
- Audio Production and Design: Music producers and sound designers create new musical works or design specific sound effects.
- Podcasts and Audiobooks: Automatically generate background music and sound effects for podcasts or audiobooks, enhancing the listener's auditory experience.
- Education and Training: In the education field, create audio that simulates real scenarios to assist language learning or professional skills training.
- Virtual Assistants and Chatbots: Provide more natural and richer voice responses for virtual assistants and chatbots, enhancing the user interaction experience.