What is GLM-4-Voice
GLM-4-Voice isZhipu AIThe end-to-end emotional voice model launched by Zhipu AI can directly understand and generate Chinese and English speech, support real-time voice dialogue, and flexibly adjust features such as emotion, intonation, speed, and dialect of the voice according to user instructions. The model consists of three parts: GLM-4-Voice-Tokenizer is responsible for converting continuous speech into discrete tokens, GLM-4-Voice-Decoder converts tokens back into continuous speech output, and GLM-4-Voice-9B is pretrained and aligned based on the GLM-4-9B model to understand and generate speech. The design of GLM-4-Voice uses end-to-end modeling to reduce information loss, improve the naturalness and fluency of voice interaction, and support low-latency real-time dialogue, providing users with a richer and more natural voice interaction experience.
Main Features of GLM-4-Voice
- Understanding and generating speech: Directly understand and generate Chinese and English speech to achieve smooth human-computer interaction.
- Emotional expression: Simulate different emotions and intonations, such as happiness, sadness, anger, fear, etc., making voice replies more natural.
- Adjust speech rate: Adjust the speech rate according to user instructions, suitable for different dialogue scenarios.
- Real-time interruption and instruction input: Support users in interrupting voice output at any time and inputting new instructions to adjust the dialogue content.
- Multilingual and dialect support: Support Chinese and English as well as multiple Chinese dialects, such as Cantonese, Chongqing dialect, Beijing dialect, etc.
- Low-latency interaction: Design a streaming thinking architecture to achieve high-quality voice dialogue with low latency.
Technical Principles of GLM-4-Voice
- End-to-end modeling: Unlike traditional cascaded solutions (ASR + LLM + TTS), GLM-4-Voice completes speech understanding and generation in a unified model, avoiding information loss.
- Audio Tokenizer: Use a supervised-trained audio Tokenizer to convert continuous speech input into discrete tokens, using a low bitrate of 12.5Hz to retain semantic information and paralinguistic features.
- Speech decoder: A speech decoder based on the Flow Matching model structure converts discrete speech tokens into continuous speech output, requiring only a minimum of 10 tokens to start generation, reducing dialogue latency.
- Pretraining and alignment: GLM-4-Voice-9B is pretrained and aligned on the basis of GLM-4-9B to understand and generate discretized speech tokens. Pretraining uses large amounts of audio and text data, giving the model powerful audio understanding and modeling capabilities.
- Streaming inference: Support streaming inference. The model can alternately output text and speech, using text as a reference to ensure the high quality of reply content, and adjust speech output in real time according to the user's voice instructions.
Project Address of GLM-4-Voice
- Product Experience：https://ai-bot.cn/sites/2005.html
- Project Official Website：zhipuai.cn/news
- GitHub Repository：https://github.com/THUDM/GLM-4-Voice
- arXiv Technical Paper：https://arxiv.org/pdf/2412.02612
Application Scenarios of GLM-4-Voice
- Intelligent assistant: In smartphones and smart home devices, as an intelligent assistant, it uses voice interaction to help users complete various tasks, such as setting reminders, checking the weather, controlling home devices, etc.
- Customer service: In customer service centers, as a virtual customer service representative, based on natural language understanding and speech synthesis technology, it provides users with consultation and problem-solving services.
- Education and Learning: In the field of education, as a language learning assistant, it helps students practice pronunciation, listening, and speaking, and provides personalized learning suggestions.
- Entertainment and Media: In the entertainment industry, it is used in speech synthesis to provide natural and expressive speech output for animations, games, audiobooks, and more.
- News and Broadcasting: Used in news broadcasting to quickly convert text news into speech and provide it to users who need voice information.