Xiaomi-CocktailASR-1 - Xiaomi's open-source target speaker speech recognition model

What is Xiaomi-CocktailASR-1

Xiaomi-CocktailASR-1 is Xiaomi's open-source target speakerspeech recognitionlarge model, based on an LLM end-to-end architecture, using reference speech as a voiceprint prompt, capable of accurately transcribing the target speaker from multi-speaker mixed speech without speech separation. The model reaches SOTA level on multi-speaker benchmarks while remaining compatible with single-speaker scenarios, and features negative sample rejection and chain-of-thought interpretable reasoning capabilities. It is open-sourced under the Apache 2.0 license.

Main features of Xiaomi-CocktailASR-1

• Target speaker speech recognition: Given reference speech and multi-speaker mixed audio, directly transcribe the target speaker's content without speech separation.
• Single-speaker compatibility: The same model can handle single-speaker scenarios, with performance comparable to mainstream single-speaker ASR, without switching models.
• Negative sample rejection: When the target speaker is not present in the audio, it outputs empty text, effectively reducing false triggers.
• Chain-of-thought reasoning: CoT mode outputs reasoning processes such as the number of speakers, gender, and voiceprint similarity, balancing interpretability and recognition accuracy.

Technical principles of Xiaomi-CocktailASR-1

• End-to-end unified architecture: The model consists of three parts: a 0.6B audio encoder improved based on Data2Vec2, a lightweight linear Adapter, and a Qwen3-8B large language model backbone. The input is concatenated in the order of "reference speech + 1 second of silence + mixed speech". After the encoder outputs frame-level features, they are aligned to the LLM hidden space via the Adapter and, together with the text Prompt, are fed into the LLM to generate the transcription of the target speaker.
• Reference speech as a conditioning prompt: The encoder uses the Data2Vec2 self-supervised masked prediction mechanism to fuse semantic and speaker information within a single network, without the need for an independent voiceprint encoder; the reference speech is treated as a conditioning Prompt, enabling the encoder to adaptively focus on the target speaker and suppress interfering speech during the feature extraction stage, avoiding from the source the error accumulation of traditional "separation + recognition" cascade systems.
• Multi-stage training with balanced multi-capabilities: Through training with a data mixture of about one million hours, four capabilities are unified: about 400,000 hours of multi-speaker data to train target extraction, about 600,000 hours of same-speaker reference-target pairs to prevent over-suppression in single-speaker scenarios, about 10,000 hours of mismatched-reference negative samples to internalize rejection capability without requiring a threshold at inference, plus CoT data to teach the model to first output a reasoning chain before giving the answer.
• Dual-mode Prompt and rejection mechanism: The standard mode uses a unified Prompt to simultaneously cover three types of tasks: single-speaker recognition, multi-speaker target recognition, and negative sample rejection. When the target is absent, the model naturally outputs empty text; the CoT mode uses an independent Prompt to trigger <think> reasoning tags, outputs intermediate steps such as the number of speakers, gender, and voiceprint similarity, and then gives <answer> the final transcription, improving interpretability and slightly reducing WER.

How to use Xiaomi-CocktailASR-1

• Install dependencies: Run pip install torch torchaudio transformers soundfile to install the required environment.
• Download the model: Download the model weight files from HuggingFace (Ease3/Xiaomi-CocktailASR-1).
• Load the model: Via AutoModel.from_pretrained("Ease3/Xiaomi-CocktailASR-1", trust_remote_code=True, torch_dtype="bfloat16").cuda().eval() load the model.
• Prepare audio: Prepare a 16k mono reference speech (target speaker voiceprint) and a single-speaker/multi-speaker audio to be recognized (non-16k will be automatically resampled).
• Single inference: Call model("target.wav", "ref_speaker.wav") to return the transcription text of the target speaker (for negative samples, empty text is automatically output, no additional parameters required).
• Chain-of-thought reasoning: Append the cot=True parameter, and the model will output the reasoning process in <think> and output the final result in <answer> .
• Batch inference: Organize the data into a 5-column TSV (utt_id, wav, text, ref_wav, ref_id), run tools/test_batch_scp.py , and specify the model path, input TSV, and output file to perform batch transcription.

Core advantages of Xiaomi-CocktailASR-1

• Multi-speaker SOTA performance: Achieves the best recognition level on multiple mainstream multi-speaker benchmarks such as AliMeeting, AMI, and LibriMix (e.g., AliMeeting Far WER 20.63%, previous SOTA was 27.5%).
• Unified single- and multi-speaker scenarios: The same model performs comparably to mainstream ASR in single-speaker scenarios, with no need to switch models based on the number of speakers.
• Negative sample rejection capability: Outputs empty text when the target speaker is absent, with a rejection rate of 68%-80%, while models such as Qwen3-ASR and StepAudio have a rejection rate of 0.
• Chain-of-thought interpretable reasoning: CoT mode outputs the reasoning process such as the number of speakers, gender, and voiceprint similarity, which both improves interpretability and slightly reduces WER.
• Concise end-to-end architecture: The reference speech is directly used as a voiceprint Prompt, eliminating the traditional speech separation module and avoiding the error accumulation of cascaded systems.
• Low usage barrier: One-line code invocation, support for automatic resampling and batch inference, and positive/negative sample demos, ready to use out of the box.

Project address of Xiaomi-CocktailASR-1

• GitHub repository：https://github.com/xiaomi-research/xiaomi-cocktailasr-1
• HuggingFace model library：https://huggingface.co/Ease3/Xiaomi-CocktailASR-1
• arXiv technical paper：https://arxiv.org/pdf/2609.11274

Comparison of Xiaomi-CocktailASR-1 with similar competitors

Application Scenarios of Xiaomi-CocktailASR-1