Covo-Audio - Tencent's Open-Source End-to-End Speech Model

What is Covo-Audio

Covo-Audio is Tencent's open-source 7-billion-parameter end-to-end speech large model that can directly process continuous audio input and generate audio output. The model's core innovations include a hierarchical three-modal speech-text interleaving architecture, intelligence-speaker decoupling technology, and native full-duplex interaction capability. The model is built onQwen2.5-7Band Whisper, achieving SOTA performance in tasks such as spoken dialogue, speech understanding, and audio understanding. As a unified-architecture speech AI, the model avoids the latency and error accumulation of traditional cascaded systems and is aGPT-4ostrong open-source alternative for speech capabilities.

Covo-Audio's Main Features

• Spoken Dialogue: Supports natural multi-turn dialogue interaction with end-to-end speech input and speech output.
• Speech Understanding: The model deeply integrates acoustic features with semantic content to achieve comprehensive parsing of high-fidelity speech signals.
• Audio Understanding: The model supports extension to non-speech scenarios and possesses comprehensive perception capabilities for generalized audio such as environmental sounds and music.
• Full-Duplex Interaction: Natively supports low-latency real-time bidirectional speech communication, allowing natural interruptions and instant responses.

Covo-Audio's Key Information and Usage Requirements

• Developer: Tencent
• Model Scale: 7 billion parameters (7B)
• Architecture Type: End-to-end unified audio language model
• Open-Source Version：Covo-Audio-Chat
• Base Model: Qwen2.5-7B (LLM backbone) + Whisper (audio encoder)
• Model Format: Safetensors, BF16 precision
• Paper：arXiv:2602.09823
• Open-Source License: Custom License (see repository for details)
• Applicable Scenarios: Research and experimental use
• Python Version: ≥ 3.11 (recommended)
• Dependency Installation: Via requirements.txt one-click installation
• Core Dependencies：Transformers、BigVGAN、huggingface-hub
• Hardware Resources: Requires a GPU that supports BF16 inference (sufficient VRAM recommended); both local deployment and cloud inference are supported

Covo-Audio's Core Advantages

• End-to-End Unified Architecture: The model breaks the traditional ASR→LLM→TTS cascade pattern, achieving direct audio-to-audio mapping, eliminating error accumulation, and significantly reducing inference latency.
• Deep Three-Modal Fusion: Through hierarchical interleaving of continuous acoustic features, discrete speech tokens, and natural language text, it establishes effective alignment between high-fidelity prosody and robust semantics.
• Intelligence and Timbre Decoupling: With multi-speaker training, the model separates dialogue intelligence from speaker characteristics, supporting flexible transfer and personalized customization of high-quality speech.
• Native Full-Duplex Capability: The model uses low-latency streaming processing to enable real-time bidirectional interaction, supporting natural interruptions and instant responses, approaching the human conversation experience.
• Open-Source Ecosystem Value: The model balances performance and cost with a 7-billion-parameter scale, and the complete open technology stack lowers the application barrier, providing an autonomous and controllable foundation solution for Chinese speech AI.

How to Use Covo-Audio

• Environment Setup: Create a Python 3.11 environment and install dependencies. Run conda create -n covoaudio python=3.11 and conda activate covoaudio, then complete the dependency installation via pip install -r requirements.txt.
• Get the Code: Clone the official GitHub repository to your local machine by running git clone https://github.com/Tencent/Covo-Audio.git and enter the project directory with cd Covo-Audio.
• Download the Model: Install the HuggingFace tools and download the pretrained weights. Run pip install huggingface-hub and hf download tencent/Covo-Audio-Chat –local-dir ./covoaudio. The model will be automatically overwritten or saved to the specified directory.
• Configure Paths: If you need to customize the model storage location, modify the model_dir and decode_load_path parameters in example.sh to match the actual paths.
• Run Inference: Execute the one-click inference script bash example.sh, or modify the audio file path in example.py to implement custom input interaction.
• Custom Usage: Replace the input audio path in example.py with your own file to conduct end-to-end voice dialogue interaction with the model.

Covo-Audio Project Address

• GitHub Repository：https://github.com/Tencent/Covo-Audio
• HuggingFace Model Library：https://huggingface.co/tencent/Covo-Audio-Chat
• arXiv Technical Paper：https://arxiv.org/pdf/2602.09823

Comparison of Covo-Audio with Similar Competing Products

Application Scenarios of Covo-Audio

• Intelligent Customer Service: The model supports end-to-end low-latency interaction and full-duplex interruption capabilities, enabling natural and smooth real-time voice question answering and personalized services with multiple voice timbres.
• Smart Hardware: The model can provide voice assistant capabilities for smart speakers, in-vehicle systems, and home control hubs, either offline or through a combination of device and cloud.
• Content Creation: Supports efficient generation of multi-character dialogue dubbing, podcast content, and real-time voice translation services.
• Education and Training: With deep understanding of voice emotion and prosodic details, it builds immersive personalized teaching interaction systems such as spoken language practice partners and virtual instructors.
• Accessibility Services: By replacing visual interfaces with natural voice interaction, it provides visually impaired groups and elderly users with convenient information access and device control without typing or touching the screen.