SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

SpeechGPT 2.0-preview - An End-to-End Real-Time Speech Interaction Model Launched by Fudan University

Author: AI小集 · Original source: AI工具集

What is SpeechGPT 2.0-preview?

SpeechGPT 2.0-preview is an anthropomorphic real-time interaction system launched by the OpenMOSS team at Fudan University. It is trained on million-hour-level Chinese speech data and adopts an end-to-end architecture, achieving a high degree of integration between speech and text modalities. The model features anthropomorphic colloquial expression and 100-millisecond-level low-latency response, and supports natural and smooth real-time interruption interaction. It can precisely control speech rate, emotion, style, and timbre, and achieve intelligent switching. SpeechGPT 2.0-preview has a variety of speech talents, such as poetry recitation, storytelling, and speaking dialects.

SpeechGPT 2.0-preview

Main Features of SpeechGPT 2.0-preview

  • Emotion and Style Control: Supports precise control of multiple emotions (such as weak and cheerful), multiple timbres (switching between male and female), and multiple styles (poetry recitation, dialect imitation), with outstanding role-playing capabilities.
  • Real-Time Interruption Interaction: 100-millisecond-level response speed supports instant interruption and continuation in natural conversations.
  • Text Capability Integration: On the basis of speech expressiveness, it retains the intelligence of text models and supports functions such as tool calling, web search, and external knowledge base access.
  • Multi-Task Compatibility: It can handle scenarios such as long document parsing and multi-turn conversations, and its compatibility with short-text tasks is not reduced due to long-context capabilities.

Technical Principles of SpeechGPT 2.0-preview

  • End-to-End Speech Modeling
    • Ultra-Low Bitrate Streaming Speech Codec: A self-developed ultra-low bitrate streaming speech Codec that can process 24khz speech input, compress speech to 75 tokens per second, support streaming input and output, and achieve real-time interaction with latency within 200ms.
    • Semantic-Acoustic Joint Modeling: Through semantic-acoustic joint modeling, it directly processes speech input and generates speech or text output, without the need for traditional cascaded ASR (automatic speech recognition) and TTS (text-to-speech) modules.
  • Speech-Text Hybrid Modeling
    • Codec Patchify: Through Codec Patchify technology, speech tokens from adjacent time steps are aggregated into unified vectors, effectively reducing the modality gap between speech and text sequences and alleviating conflict issues in cross-modal modeling.
    • Multi-Stage Training Pipeline: Includes modality adaptation pretraining, cross-modal instruction fine-tuning, and chain-of-modality fine-tuning, balancing text capabilities and speech capabilities, and preventing the model's intelligence from declining when learning speech capabilities.
  • Speech-Text Alignment Pretraining: Through sufficient speech-text alignment pretraining, the model can "emerge" with generalization of speech styles, for example, controlling speech rate without speech rate adjustment data, or imitating unseen character tone styles.
  • Model Architecture Design
    • Speech-Text Joint Modeling: The speech-text LLM simultaneously inputs and outputs speech and text representations, and the LLM's hidden states are used for both speech and text decoding tasks.
    • Autoregressive Language Model with Multiple Decoding Heads: An autoregressive language model with multiple decoding heads is designed as a patch decoder, which can decode step by step in an autoregressive manner, generating multiple RVQ codec tokens for one time step each time and outputting speech.

Project Address of SpeechGPT 2.0-preview

Application scenarios of SpeechGPT 2.0-preview

  • Intelligent assistant: Can be used as an intelligent assistant in fields such as customer service, education, or healthcare, providing services such as real-time spoken language practice and emotional companionship.
  • Content creation: Automatically generate audiobooks, poetry recitations, or dialect content, enriching multimedia creation formats.
  • Accessible communication: Provide real-time speech-to-text and synthesis services for people with hearing or speech impairments.