What is Higgs Audio V2
Higgs Audio V2 is an open-source speech large model developed by Li Mu and his team at Boson AI. Trained on over 10 million hours of audio data, it features multilingual dialogue generation, automatic prosody adjustment,voice cloningand song synthesis. The model can simulate natural and fluent multi-person conversations, automatically matching the speaker's emotions and tone, and supports low-latency real-time voice interaction. It supports zero-shot voice cloning; users only need to provide a short voice sample to replicate the voice characteristics of a specific person, and it can synthesize singing. Higgs Audio V2 can simultaneously generate speech and background music, providing strong support for audio content creation.
Main features of Higgs Audio V2
- Multilingual dialogue generation: Supports multilingual dialogue generation, can simulate multi-person interactive scenarios, automatically matches the speaker's emotion and energy level, making the conversation natural and fluent.
- Automatic prosody adjustment: In long-text reading, it can automatically adjust speed, pauses, and intonation based on the content without manual intervention, generating natural and fluent speech.
- Voice cloning and song synthesis: Users only need to provide a short voice sample, and the model can achieve zero-shot voice cloning, replicate the voice characteristics of a specific person, and enable the cloned voice to hum a melody.
- Real-time voice interaction: Supports low-latency response, can understand user emotions and express them emotionally, providing an interaction experience close to human.
- Synchronized generation of speech and background music: Can simultaneously generate speech and background music, realizing the creative process of "write a song and sing it."
Technical principles of Higgs Audio V2
-
AudioVerse dataset: Developed an automated annotation pipeline, combining multiple speech recognition models, sound event classification models, and a self-developed audio understanding model to clean and annotate 10 million hours of audio data.
-
Unified audio tokenizer: Trained a unified audio tokenizer from scratch that can simultaneously capture semantic and acoustic features.
-
DualFFN architecture: Significantly enhances the large language model's ability to model acoustic tokens with almost no increase in computational overhead.
-
Zero-shot voice cloning: The model incorporates in-context learning, enabling zero-shot voice cloning through simple prompts (such as a short reference audio sample) and matching the speaking style.
Project address of Higgs Audio V2
- Github repository：https://github.com/boson-ai/higgs-audio
- Online experience Demo：https://huggingface.co/spaces/smola/higgs_audio_v2
Application scenarios of Higgs Audio V2
-
Real-time voice interaction: Suitable for scenarios such as virtual anchors and real-time voice assistants, providing natural interaction with low latency and emotional expression.
-
Audio content creation: Can generate natural dialogue and narration, providing strong support for audiobooks, interactive training, and dynamic storytelling.
-
Entertainment and creative fields: The voice cloning feature can replicate the voices of specific people, opening up new possibilities in entertainment and creative fields.