What is MiDashengLM
MiDashengLM is Xiaomi's open-source efficient large model for sound understanding, with the specific parameter version being MiDashengLM-7B. The model is built on the Xiaomi Dasheng audio encoder and the Qwen2.5-Omni-7B Thinker decoder, using a general audio description alignment strategy to achieve unified understanding of speech, environmental sounds, and music. The model delivers excellent performance and high inference efficiency, with first-token latency only 1/4 that of state-of-the-art models, and supports large-scale parallel processing. The model's training data is fully open-sourced, supports academic and commercial use, and is suitable for scenarios such as smart cockpits and smart homes, driving the upgrade of multimodal interaction experiences.
Main Features of MiDashengLM
- Audio Captioning: Converts audio content (including speech, environmental sounds, music, etc.) into natural language descriptions, helping users quickly understand audio information.
- Audio Classification: Identifies specific categories in audio (such as speech, environmental sounds, music, etc.), used in scenarios such as environmental sound recognition and music classification.
- Automatic Speech Recognition (ASR): Converts speech into text, supports multiple languages, and is widely used in scenarios such as voice assistants and smart cockpits.
- Audio Question Answering: Answers related questions based on input audio content, applicable to environmental sound question answering, music question answering, etc. in smart cockpits.
- Multimodal Interaction: Combines audio with other modalities (such as text and images) for comprehensive understanding, enhancing the interaction experience of smart devices.
Technical Principles of MiDashengLM
- Model Architecture：
- Audio Encoder: Based on the Xiaomi Dasheng audio encoder, responsible for converting input audio signals into high-dimensional feature representations. The Dasheng encoder performs excellently in audio understanding tasks, and can extract rich semantic information when processing non-speech audio (such as environmental sounds and music).
- Decoder: Based on the Qwen2.5-Omni-7B Thinker autoregressive decoder, responsible for converting the features extracted by the audio encoder into natural language descriptions. The decoder supports multiple tasks, including audio captioning, audio question answering, and speech recognition, etc.
- Training Strategy：
- General Audio Description Alignment: Based on the general audio description alignment strategy, it avoids the limitations of traditional ASR transcription methods, and through non-monotonic global semantic mapping, forces the model to learn deep semantic associations of audio scenes, achieving unified understanding of speech, environmental sounds, and music.
- Multi-Expert Analysis: The training data is generated based on a multi-expert analysis pipeline, including fine-grained annotations of speech, vocals, music, and environmental acoustics, and then uses the DeepSeek-R1 reasoning large model to synthesize unified descriptions.
- Dataset: Trained on public datasets, covering multiple domains such as speech, environmental sounds, and music, with a total duration of over 1 million hours. The original labels of the training data are discarded during pretraining, and only new rich text description labels are used, forcing the model to learn richer and more comprehensive sound information.
- Inference Efficiency Optimization：
- Efficient Inference: By optimizing the audio encoder design, the output frame rate is reduced from 25Hz in Qwen2.5-Omni to 5Hz, significantly reducing computational load and improving inference efficiency.
- Massively parallel processing: Supports larger batch processing (batch size=512). When processing 30 seconds of audio on an 80GB GPU and generating 100 tokens, MiDashengLM achieves more than 20 times the throughput speed of Qwen2.5-Omni-7B.
MiDashengLM project address
- GitHub repository：https://github.com/xiaomi-research/dasheng-lm
- HuggingFace model library：https://huggingface.co/mispeech/midashenglm-7b
- Technical paper：https://github.com/xiaomi-research/dasheng-lm/blob/main/technical_report/MiDashengLM_techreport.pdf
- Online demo：https://huggingface.co/spaces/mispeech/MiDashengLM-7B
MiDashengLM application scenarios
- Smart cockpit: Based on voice assistants and environmental sound recognition, it improves driving safety and interactive experience.
- Smart home: Uses voice control and environmental sound monitoring to achieve convenient home automation.
- Voice assistant: Provides multilingual speech recognition and intelligent dialogue to meet users' diverse needs.
- Audio content creation and annotation: Automatically generates audio descriptions and tags to improve content creation efficiency.
- Education and learning: Assists users in language and music learning, providing pronunciation feedback and theoretical guidance.