SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

GLM-TTS - Zhipu's open-source AI text-to-speech model, supporting voice cloning

Author: AI小集 · Original source: AI工具集

What is GLM-TTS

GLM-TTS is an open-sourceAI text-to-speechmodel from Zhipu, featuringvoice cloning, multi-emotion expression, high-precision text understanding, and high-quality speech output. It adopts a two-stage generation paradigm, achieving efficient speech synthesis through semantic modeling and acoustic modeling. GLM-TTS supports dialect cloning, fine-grained pronunciation control, and multiple emotional expressions, and is suitable for scenarios such as intelligent voice assistants, audiobooks, education, entertainment, customer service, accessibility assistance, advertising, news, smart homes, and virtual anchors. With low character error rates and high naturalness, GLM-TTS provides users with a flexible, efficient, and personalized voice interaction experience, promoting the widespread application of speech synthesis technology across multiple fields. GLM-TTS has been open-sourced on platforms such as GitHub, Hugging Face, and ModelScope.

GLM-TTS

Main Features of GLM-TTS

  • Voice Cloning: GLM-TTS can quickly clone the timbre of a specific speaker, achieving highly similar speech synthesis with only a small amount of audio data, and supports voice cloning in multiple languages and dialects.
  • Multi-Emotion Cloning: It automatically matches the corresponding speech emotion based on the emotion of the text content, supports multiple emotional expressions such as happiness, sadness, and anger, and enhances the naturalness and expressiveness of speech synthesis.
  • High-Precision Text Understanding: It has powerful text understanding capabilities, can accurately process text content, reduces character error rates, and ensures the accuracy and coherence of synthesized speech.
  • Dialect and Special Speech Synthesis: It supports the synthesis of multiple dialects and special speech, such as Sichuan dialect and Northeastern dialect, adapting to the needs of different languages and cultural backgrounds.
  • Fine-Grained Pronunciation Control: Through phoneme-level input (Phoneme-in) technology, it solves pronunciation issues with polyphonic characters and rare characters, improving pronunciation accuracy and controllability.
  • High-Fidelity Speech Output: Based on the self-developed 2D-Vocos vocoder, it generates high-quality, high-fidelity speech waveforms, supports high sampling rate output, and improves sound quality performance.

Technical Principles of GLM-TTS

  • Two-Stage Generation Paradigm: GLM-TTS adopts a two-stage generation paradigm, divided into semantic modeling (Text-to-Token) and acoustic modeling and waveform reconstruction (Token-to-Wav). In the first stage, an autoregressive model converts text into a semantic Token sequence, ensuring the accuracy and coherence of the content; in the second stage, a Conditional Flow-matching model predicts the mel spectrogram, which is converted into high-quality speech waveforms through the 2D-Vocos vocoder.
  • Multi-Reward Reinforcement Learning: GLM-TTS introduces a multi-reward reinforcement learning mechanism based on the GRPO algorithm framework, integrating multi-dimensional rewards such as character error rate (CER), similarity (Sim), emotion, and paralinguistic features (such as laughter). It optimizes the training process through dynamic sampling and gradient clipping strategies, significantly improving the emotional expressiveness and anthropomorphic quality of speech.
  • Fine-Grained Pronunciation Control (Phoneme-in): GLM-TTS provides Phoneme-in technology, which provides target phonemes for polyphonic characters and rare characters through a dynamically controllable dictionary and mixed input forms, achieving precise pronunciation control. During inference, the system combines the phoneme sequence with the text as input to the model, preserving the prosody of the text while ensuring accurate pronunciation.
  • Premium Voice Customization (LoRA): GLM-TTS adopts an optimized LoRA fine-tuning paradigm, requiring fine-tuning of only about 15% of model parameters. Combined with a small amount of high-quality audio data, it achieves timbre fidelity and naturalness comparable to full-parameter fine-tuning, greatly reducing the development cost and deployment threshold for timbre customization and improving timbre generalization ability and cross-scenario stability.
  • Data Processing and Feature Extraction: GLM-TTS has built a complete data processing pipeline, including steps such as speech standardization, background sound separation and noise reduction, speaker separation and concatenation, WER filtering, punctuation optimization, and feature extraction. It extracts pure speech and high-quality features from heterogeneous audio, providing reliable data support for model training.
  • Model Architecture Optimization: GLM-TTS optimizes the Speech Tokenizer, improving the token bitrate and vocabulary size, introducing a pitch estimation module (PE), removing the causal convolution restriction, and optimizing pitch modeling accuracy; at the same time, the 2D-Vocos vocoder improves the parsing accuracy of spectral features and sound quality performance through 2D convolution and DiT-like residual connections, enhancing the model's adaptability to complex vocal timbres.

GLM-TTS Project Address

  • GitHub Repository：https://github.com/zai-org/GLM-TTS
  • HuggingFace Model Library：https://huggingface.co/zai-org/GLM-TTS

How to Use GLM-TTS

  • Online Experience: Visit https://audio.z.ai/ , upload text or a speech prompt to quickly generate speech.
  • API Call: ViaZhipu AI Open Platformobtain an API key, send requests according to the documentation, and send text or timbre requirements to the GLM-TTS server to obtain synthesized speech.
  • Local Deployment: Download model resources from GitHub, Hugging Face, or ModelScope, deploy them in a local GPU environment, and use them for secondary development or customized use.

Application Scenarios of GLM-TTS

  • Intelligent Voice Assistant: GLM-TTS provides natural and fluent voice feedback for intelligent voice assistants, supports multiple languages and emotional expression, and generates voice interactions that fit the scenario according to user instructions, improving user experience.
  • Audiobooks and Audio Content Creation: GLM-TTS can quickly generate voice content in different styles and emotions, supports switching between multiple timbres and dialect reading, and meets diverse audio creation needs such as audiobooks and podcasts.
  • Education and Training: Through fine-grained pronunciation control, GLM-TTS helps learners correct the pronunciation of polyphonic and uncommon characters, supports multilingual and dialect teaching, and improves education quality and efficiency.
  • Entertainment and Games: GLM-TTS generates speech with dialects and emotions for game characters and entertainment content, enhancing the immersion and fun of games and entertainment.
  • Customer Service and Intelligent Interaction: GLM-TTS adjusts the speech style according to user emotions and generates gentle or patient voice responses, improving the interaction experience and customer satisfaction of intelligent customer service.