Higgs Audio v2 is a text-to-audio foundation model released by Boson AI, pretrained on over 10 million hours of audio data and diverse text data. According to the official introduction, the model performs outstandingly in expressive audio generation without post-training or fine-tuning: on the "Emotions" and "Questions" categories of EmergentTTS-Eval, its win rates against "gpt-4o-mini-tts" are 75.7% and 55.7%, respectively; on traditional TTS benchmarks such as Seed-TTS Eval and the Emotional Speech Dataset (ESD), it achieved state-of-the-art performance at the time. The model also demonstrates capabilities rarely seen in previous systems, including multilingual natural multi-speaker dialogue generation, automatic prosodic adaptation during narration, humming melodies using cloned voices, simultaneous generation of speech and background music, and support for multilingual real-time translation.
This repository, train-higgs-audio, is the training repository for Higgs Audio v2, providing data processing, training, and generation scripts. The repository includes tutorial videos and screen recordings showing how to use the training and generation scripts. The current version mainly implements single-speaker training; the multi-speaker training part is mentioned in the materials but specific steps are not elaborated. Training for new languages is an experimental feature, and details of DDP support should refer to DDP_training.sh. The repository optimizes input parameters, removes unnecessary misleading parameters, adopts official data classes, and supports LoRA training, enabling training with just 16G of memory. It also provides a mini training set for use.
In terms of usage, the training process includes: preparing audio and text data that meet the requirements, processing it according to the ms-swift data format, running scripts to obtain data in metadata.json format, modifying various parameters before training (data path, model path, number of training epochs, etc.), and then executing training. LoRA fine-tuning requires the --use_lora parameter. The materials note that when using LoRA fine-tuning for a new voice, there may be cases where output cannot be generated normally. This issue was found during Vietnamese transfer fine-tuning, and the cause is still unclear; based on past experience, when training a model to learn knowledge it has never encountered before, it is best to use full fine-tuning (--use_lora False).
In terms of generation, the repository provides multiple examples (located in the examples directory), including zero-shot voice cloning (generating a voice similar to the reference audio), single-speaker intelligent speech generation (when no reference voice is specified, the model, based on the transcribed textThis decision voice), multi-speaker dialogue intelligent speech generation, and multi-speaker dialogue voice cloning. The generation script automatically uses cuda:0 when CUDA availability is detected, and the device can be changed via --device_id. For optimal performance, the official recommendation is to run generation examples on a GPU with at least 24GB of VRAM.
On the technical side, Higgs Audio v2 adopts the "generation variant" in the architecture diagram. Its performance is driven by three key technical innovations: first, an automated annotation pipeline that uses multiple ASR models, a sound event classification model, and a self-developed audio understanding model to clean and annotate 10 million hours of audio data (called AudioVerse); the self-developed understanding model is fine-tuned based on Higgs Audio v1 Understanding; second, training a unified audio tokenizer from scratch that captures both semantic and acoustic features simultaneously; third, proposing the DualFFN architecture to enhance the LLM's ability to model acoustic tokens with minimal computational overhead.
Performance evaluation covers Seed-TTS Eval, ESD, EmergentTTS-Eval, and multi-speaker evaluation. Zero-shot TTS uses reference text, reference audio, and target text prompts, adopting the standard metrics of Seed-TTS Eval and ESD; EmergentTTS-Eval reports the win rate against "gpt-4o-mini-tts" (alloy voice), with Gemini 2.5 Pro as the judging model, and "†" indicates the use of the strong prompting method from the paper. The multi-speaker evaluation benchmark contains three subsets: two-speaker-conversation (1000 two-person synthetic dialogues, with two fixed reference audios, evaluating dual-voice cloning), small talk (no ref) (250 short dialogues, without fixed reference audio, evaluating the ability to automatically assign voices), and small talk (ref) (250 shorter dialogues, with reference audio included in the context). It reports word error rate (WER) and the geometric mean of intra-speaker similarity and inter-speaker difference, and compares with MoonCast and nari-labs/Dia-1.6B-0626.
Limitations and licensing: currently only single-speaker training is implemented; the specific implementation of multi-speaker training is not provided; LoRA fine-tuning for new voices may fail, and full fine-tuning is recommended; generation requires a GPU with at least 24GB of VRAM (official recommendation); boson_multimodal/The audio_processing/ directory contains code from third-party repositories (mainly from xcodec); see the LICENSE in that directory for licensing information.
