What Is Hibiki
Hibiki is an open-source decoder model from Kyutai Labs for simultaneous speech translation, capable of translating speech from one language into speech or text in another language in real time. Hibiki is based on a multi-stream language model architecture, synchronously processing source speech and target speech, and jointly generating text and audio tokens, enabling speech-to-speech (S2ST) andspeech-to-text(S2TT) translation capabilities. Hibiki uses a weakly supervised learning approach, identifying the best word-level latency based on the perplexity of a text translation system, and creating aligned synthetic data for training. The Hibiki model performs excellently in French-to-English translation tasks, with high translation quality, speaker fidelity, and naturalness, supports batch translation and real-time on-device deployment, and demonstrates strong practical potential.
Hibiki's Main Features
- Real-Time Speech-to-Speech Translation (S2ST): Translates speech in one language into speech in another language in real time, preserving the speaker's timbre and intonation.
- Real-Time Speech-to-Text Translation (S2TT): Translates speech into text in the target language in real time, providing more flexible usage scenarios.
- Low-Latency Translation: Based on accumulating contextual information in real time, it generates translated content chunk by chunk, with extremely low latency, approaching the level of human interpretation.
- High Fidelity: The generated speech is natural and fluent, highly similar to the source speech in timbre and intonation, and the user experience is close to professional human interpretation.
- Supports Batch and Real-Time Deployment: The inference process is simple, supports batch processing and real-time on-device deployment, and is suitable for large-scale applications.
Hibiki's Technical Principles
- Multi-Stream Language Model Architecture：
- Synchronous Processing: Simultaneously receives source speech and generates target speech, jointly modeling the two audio streams based on a multi-stream architecture.
- Text and Audio Tokens: The model predicts a hierarchical structure of text and audio tokens, enabling speech-to-text and speech-to-speech translation.
- Causal Audio Codec: Uses a pretrained causal audio codec (such as Mimi) to encode speech into low-frame-rate discrete tokens, supporting real-time streaming processing.
- Weakly Supervised Learning and Context Alignment：
- Synthetic Data Generation: Based on transcribing the translated monolingual audio and re-synthesizing the target speech, generates aligned synthetic data.
- Context Alignment: Uses the perplexity of an off-the-shelf text translation system to compute word-level alignment, ensuring that the generation of the target speech is synchronized with the context of the source speech.
- Silence Insertion and Alignment-Aware TTS: Based on inserting silence or re-synthesizing the target speech with an alignment-aware TTS model, ensures that the latency of the target speech meets the requirements of real-time translation.
- Speaker Similarity and Classifier-Free Guidance：
- Speaker Similarity Tagging: Tags the training data with speaker similarity classification, avoiding data filtering while prioritizing high-similarity samples during inference.
- Classifier-Free Guidance: Adjusts the weights of conditional labels, enhancing the model's control over speaker similarity and further improving speech fidelity.
- Efficient Inference Process：
- Temperature Sampling: Uses temperature sampling technology, combined with a causal audio codec, to enable streaming input and output.
- Batch Processing and Real-Time Deployment: The inference process is simple and efficient, supports batch processing and real-time on-device deployment, and is suitable for large-scale application scenarios.
Hibiki's project address
- GitHub repository：https://github.com/kyutai-labs/hibiki
- HuggingFace model library：https://huggingface.co/collections/kyutai/hibiki
- arXiv technical paper：https://arxiv.org/pdf/2502.03382
Hibiki's application scenarios
- International conferences: Real-time translation of speeches in different languages, helping attendees understand the content instantly.
- Online education: Real-time translation of teachers' lecture speech, making it convenient for students to learn without barriers.
- Travel: Real-time translation of tour guide explanations or conversations with locals, enhancing the travel experience.
- News interviews: Helps reporters quickly translate interview content and improves reporting efficiency.
- Customer service: Enables multilingual customer service communication and improves customer satisfaction.