What is VITA-Audio
VITA-Audio is an open-source end-to-end multimodal speech large model, featuring low latency and fast inference speed. Through a lightweight multimodal cross-token prediction (MCTP) module, it can generate audio output in the first forward pass, greatly shortening the time to generate the first audio token block and significantly reducing latency in streaming scenarios. It adopts a four-stage progressive training strategy and, at the 7-billion-parameter scale, achieves inference speeds 3 to 5 times faster than open-source models of similar scale, while also performing excellently inspeech recognition（ASR）、text-to-speech(TTS) and spoken question answering (SQA) and other task benchmarks.
Main features of VITA-Audio
- Real-time conversation capability: VITA-Audio is a multimodal large language model that can generate audio output in the first forward pass, enabling extremely low-latency real-time conversation. Through the lightweight multimodal cross-token prediction (MCTP) module, the model can efficiently generate multiple audio tokens in a single forward pass, significantly reducing the latency of generating the first audio in streaming scenarios.
- Efficient inference acceleration: At the 7-billion-parameter scale, VITA-Audio achieves 3 to 5 times inference acceleration. Thanks to its unique four-stage progressive training strategy, it achieves efficient model inference while minimizing speech quality loss.
- Multimodal interaction capability: VITA-Audio can understand and process data in multiple modalities, including audio and text. It has broad application potential in multimodal interaction scenarios, such as voice assistants and intelligent customer service, and can provide richer and more natural interaction experiences.
- Speech generation and recognition: In multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks, VITA-Audio significantly outperforms open-source models of similar model scale. It has high accuracy and efficiency in speech generation and recognition, and can meet speech interaction needs in different scenarios.
Technical principles of VITA-Audio
- Multimodal cross-token prediction (MCTP) module: It is one of the key innovations of VITA-Audio. The module can efficiently generate multiple audio tokens in a single model forward pass, significantly reducing the latency of generating the first audio in streaming scenarios and enabling extremely low-latency real-time conversation.
- TiCodec module: Used to encode continuous speech waveforms into discrete tokens and decode discrete tokens back into waveforms. This allows speech and text to be uniformly modeled in the same sequence space.
- Non-autoregressive (NAR) and autoregressive (AR) decoders: The NAR decoder predicts the distribution of the entire speech token sequence at once, which is fast but relatively less accurate; the AR decoder relies on previously generated tokens, producing higher generation quality but slower speed. VITA-Audio combines both: it first uses NAR to generate a preliminary speech token distribution, then AR refines it, balancing speed and quality.
- Four-stage progressive training strategy
- Stage 1: Vision-language alignment: By training a vision adapter and fine-tuning the model with descriptive captions and visual question answering data, it establishes strong visual capabilities.
- Stage 2: Audio input fine-tuning: Train the audio encoder using speech transcription paired data, then fine-tune with speech question answering data, enabling the model to understand and respond to audio input.
- Stage 3: Audio output fine-tuning: Train the audio decoder to achieve end-to-end speech output without the need for an external TTS module.
- Stage 4: Multimodal Instruction Fine-Tuning: Further optimize the model's performance in multimodal tasks and enhance its ability to process inputs from different modalities.
VITA-Audio's project address
- GitHub repository：https://github.com/VITA-MLLM/VITA-Audio
- HuggingFace model library：https://huggingface.co/collections/VITA-MLLM/vita-audio
- arXiv technical paper：https://arxiv.org/pdf/2505.03739
VITA-Audio's application scenarios
- Smart home control: VITA-Audio can understand voice commands and control smart devices in the home, such as lights, temperature, security systems, etc.
- Intelligent customer service: In online customer service scenarios, VITA-Audio can simultaneously understand the user's text descriptions, uploaded images, or voice messages, and quickly and accurately answer questions.
- Educational assistance: In online education platforms or learning applications, VITA-Audio can assist students in learning, for example by analyzing learning materials uploaded by students (such as textbook images, instructional videos, etc.) and, combined with students' questions, providing services such as explanations of knowledge points and answers to exercises.
- Medical assistance: VITA-Audio can help doctors interpret medical images, analyze medical record data, and, combined with the latest medical research literature, provide assistance in diagnosis and suggestions for treatment plans.
- Content creation: At media companies or self-media platforms, VITA-Audio can assist content creators in generating articles, video scripts, etc., and by understanding the creative theme and style requirements, provide creative inspiration and writing suggestions.