SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

VoxCPM - A Speech Generation Model Jointly Launched by ModelBest and Tsinghua University

Author: AI小集 · Original source: AI工具集

What is VoxCPM

VoxCPM is a 0.5B parameter speech generation model jointly developed by ModelBest and Tsinghua University Shenzhen International Graduate School. It has reached the industry's top level in terms of naturalness, timbre similarity, and prosodic expressiveness in speech synthesis. VoxCPM adopts an end-to-end diffusion autoregressive architecture, generating continuous speech representations directly from text, breaking through the limitations of traditional discrete tokenization. Through hierarchical language modeling and finite-state quantization constraints, it achieves implicit decoupling of semantics and acoustics, significantly improving speech expressiveness and generation stability. VoxCPM supports zero-shotvoice cloning, requiring only a reference audio clip to accurately replicate the speaker's timbre, accent, emotional tone, and other characteristics, generating highly realistic speech. Its inference efficiency is extremely high: on an NVIDIA RTX 4090 GPU, the real-time factor (RTF) is as low as 0.17, meeting the needs of real-time applications. VoxCPM supports bilingual Chinese-English voice replication, can synthesize audio for formulas and symbols, and enables custom pronunciation correction.

VoxCPM

Main Features of VoxCPM

  • Context-Aware Speech Generation: VoxCPM can deeply understand text content, infer and generate appropriate prosody based on the semantics of the text, and output highly expressive, fluent, and natural speech. It can autonomously adjust the speaking style based on the text content, and is trained on a massive 1.8 million-hour bilingual corpus to generate highly fitting personalized voice expressions.
  • Zero-Shot Voice Cloning: With just a short reference audio clip, VoxCPM can achieve precise zero-shot voice cloning. It can perfectly replicate the speaker's timbre and capture subtle features such as accent, emotional tone, rhythm, and pauses, creating a highly faithful and natural imitation voice.
  • Efficient Synthesis: VoxCPM supports streaming synthesis, and on a consumer-grade NVIDIA RTX 4090 GPU, its real-time factor (RTF) is as low as 0.17, easily meeting the needs of real-time applications.
  • Multilingual Support: VoxCPM is primarily trained for English and Chinese, and can generate high-quality bilingual Chinese-English speech, suitable for various language environments and application scenarios.
  • Flexible Text Input Methods: VoxCPM supports multiple text input methods, including plain text input and phoneme input. Users can choose different input modes as needed to achieve more precise pronunciation control.
  • Powerful Speech Processing Capabilities: VoxCPM can handle complex text content, including special text such as formulas and symbols, and generate corresponding speech output. It supports custom pronunciation correction, allowing users to achieve specific pronunciation needs through phoneme tag substitution.

Technical Principles of VoxCPM

  • End-to-End Diffusion Autoregressive Architecture: VoxCPM adopts an end-to-end diffusion autoregressive architecture, directly generating continuous speech representations from text, breaking through the limitations of traditional discrete tokenization and handling the continuity of speech more naturally.
  • Hierarchical Language Modeling and FSQ Constraints: Through hierarchical language modeling and finite-state quantization (FSQ) constraints, VoxCPM achieves implicit semantic-acoustic decoupling, significantly enhancing speech expressiveness and generation stability.
  • Local Audio Encoding Module (LocEnc Module): The module is responsible for encoding the input text, extracting the semantic information of the text, and converting it into an intermediate representation suitable for speech generation.
  • Text-Semantic Language Model (Text-Semantic LM, TSLM): TSLM is responsible for modeling the semantics of text, generating semantic representations related to the text content, and providing a semantic foundation for subsequent speech generation.
  • Residual Acoustic Language Model (Residual Acoustic LM, RALM): RALM further refines acoustic features on the basis of TSLM, adding acoustic details to make the generated speech more natural and realistic.
  • Local Diffusion Generation Module (LocDiT Module): The LocDiT module generates continuous speech features through a diffusion process, fusing semantic and acoustic information to ultimately generate high-quality speech waveforms.
  • Causal VAE Codec: Used to compress the original audio waveform into a low-frame-rate latent space and reconstruct the generated speech representations back into waveform signals, ensuring that the generated speech has good quality and stability.

VoxCPM Project Address

  • Github Repository： https://github.com/OpenBMB/VoxCPM/
  • Hugging Face Model Library:  https://huggingface.co/openbmb/VoxCPM-0.5B
  • Online Experience Demo: https://huggingface.co/spaces/OpenBMB/VoxCPM-Demo

VoxCPM Application Scenarios

  • Voice Assistant: VoxCPM can provide natural and fluent speech synthesis capabilities for intelligent voice assistants, enabling interaction with users using speech closer to that of humans and improving the user experience.
  • Audiobooks: It can convert text content into high-quality speech, making it suitable for producing audiobooks, audio novels, etc., bringing users a more vivid listening experience.
  • Voice Broadcasting: It can be used in scenarios such as weather forecasts, news broadcasts, and traffic information broadcasts to generate clear and natural voice broadcast content, improving the efficiency and accuracy of information delivery.
  • Voice Cloning: VoxCPM's zero-shot voice cloning capability can be used to create personalized voices, for example, giving unique voice characteristics to virtual characters, intelligent customer service, etc., enhancing their realism and recognizability.
  • Education Sector: In scenarios such as language learning and online education, VoxCPM can generate standard speech examples to help learners better imitate and learn pronunciation.
  • Entertainment Industry: In entertainment fields such as games, animation, and film and television, VoxCPM can generate the voices of various characters, enriching the expressiveness and appeal of the content.