SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Xiaomi-MiMo-Audio - Xiaomi's Open-Source End-to-End Speech Large Model

Author: AI小集 · Original source: AI工具集

What is Xiaomi-MiMo-Audio

Xiaomi-MiMo-Audio is Xiaomi's first open-source native end-to-end speech large model. Based on an innovative pre-training architecture and over a hundred million hours of training data, the model achieves few-shot generalization capability based on In-Context Learning (ICL) in the speech domain for the first time, breaking the bottleneck of relying on large-scale annotated data in the speech domain. Xiaomi-MiMo-Audio significantly surpasses open-source models of the same parameter count across multiple standard evaluation benchmarks, achieving the best performance at 7B. On the standard test set of the audio understanding benchmark MMAU, it surpasses Google's Gemini-2.5-Flash, and on the Big Bench Audio S2T task, an audio complex reasoning benchmark, it surpasses OpenAI's GPT-4o-Audio-Preview.

Xiaomi has open-sourced the pre-trained model MiMo-Audio-7B-Base and the instruction-tuned model MiMo-Audio-7B-Instruct, as well as a Tokenizer model with 1.2B parameters, supporting audio reconstruction and audio-to-text tasks.

Xiaomi-MiMo-Audio

Main Features of Xiaomi-MiMo-Audio

  • Few-Shot Generalization Capability: For the first time in the speech domain, it achieves few-shot generalization based on In-Context Learning (ICL), enabling rapid adaptation to new tasks, witnessing the "GPT-3 moment" of the speech domain.
  • Cross-Modal Alignment Capability: Post-training elicits cross-modal alignment capabilities such as IQ, EQ, expressiveness, and safety, with speech dialogues presenting an extremely high level of human-likeness in naturalness, emotional expression, and interaction adaptation.
  • Speech Understanding and Generation: It significantly surpasses open-source models of the same parameter count across multiple standard evaluation benchmarks including general speech understanding and dialogue, achieving the best performance at 7B, and also surpasses some closed-source speech models.
  • Audio Complex Reasoning: It performs excellently on the Big Bench Audio S2T task, a benchmark for audio complex reasoning, demonstrating strong audio complex reasoning capability.
  • Speech Continuation Capability: The pre-trained model MiMo-Audio-7B-Base is currently the first speech model in the open-source field with speech continuation capability.
  • Supports Hybrid Thinking: It is the first open-source model to introduce Thinking simultaneously into both speech understanding and speech generation processes, supporting hybrid thinking.
  • Audio-to-Text Task: The Tokenizer model supports the audio-to-text (A2T) task, covering over ten million hours of speech data.

Technical Principles of Xiaomi-MiMo-Audio

  • Innovative Pre-Training Architecture: It adopts an innovative pre-training architecture, trained on over a hundred million hours of training data, enabling the model to better process speech data.
  • Few-Shot Generalization Capability: For the first time in the speech domain, it achieves few-shot generalization capability based on In-Context Learning (ICL), enabling rapid adaptation to new tasks with just a few samples.
  • Cross-Modal Alignment Capability: Post-training further elicits the model's cross-modal alignment capabilities such as IQ, EQ, expressiveness, and safety, enabling speech dialogues to reach an extremely high level of human-likeness in naturalness, emotional expression, and interaction adaptation.
  • Lossless Compression Pre-Training: Through speech lossless compression pre-training, cross-task generalization is achieved, proving "emergent" behavior in the speech domain.
  • Tokenizer Model: It adopts a Transformer-architecture Tokenizer model with 1.2B parameters, trained from scratch, covering over ten million hours of speech data, supporting audio reconstruction tasks and audio-to-text (A2T) tasks.
  • Lightweight Post-training: Through lightweight post-training (SFT), the model's performance is further optimized, enabling it to excel in speech understanding and generation.
  • Hybrid Thinking Mechanism: The Thinking mechanism is introduced into both speech understanding and speech generation processes, supporting hybrid thinking and enhancing the model's complex reasoning capabilities.

Project Address of Xiaomi-MiMo-Audio

  • Project Official Website：https://xiaomimimo.github.io/MiMo-Audio-Demo/
  • Github Repository：https://github.com/XiaomiMiMo/MiMo-Audio
  • HuggingFace Model Library
    • MiMo-Audio-7B-Base：https://huggingface.co/XiaomiMiMo/MiMo-Audio-7B-Base
    • MiMo-Audio-7B-Instruct：https://huggingface.co/XiaomiMiMo/MiMo-Audio-7B-Instruct
    • Tokenizer：https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer
  • Technical Paper：https://github.com/XiaomiMiMo/MiMo-Audio/blob/main/MiMo-Audio-Technical-Report.pdf

Application Scenarios of Xiaomi-MiMo-Audio

  • Voice Interaction: It can be used for intelligent voice assistants, providing a more natural and smarter voice interaction experience, and supporting conversations in multiple languages and dialects.
  • Speech Generation: It can generate high-quality speech content and is suitable for scenarios such as audiobooks, voice announcements, and voice navigation.
  • Speech-to-Text: It supports speech-to-text (A2T) tasks and can be applied to scenarios such as meeting transcription, voice input, and voice search.
  • Audio Content Creation: It helps content creators generate audio scripts or speech content, improving creation efficiency.
  • Emotional Expression: It displays rich emotional expression in voice conversations and is suitable for scenarios requiring emotional interaction, such as emotional companion robots and customer service systems.
  • Speech Recognition and Understanding: It performs excellently in audio understanding benchmark tests and can be used in scenarios such as speech recognition and voice command control.