What is Dolphin
Dolphin is a speech large model for Eastern languages jointly launched by the Speech and Audio Technology Laboratory of the Department of Electronic Engineering at Tsinghua University and Speechocean. It supports speech recognition for 40 Eastern languagesspeech recognition, and the Chinese languages cover 22 dialects (including Mandarin), and it can accurately recognize the language characteristics of different regions. The total duration of the model's training data is 212,000 hours, including 138,000 hours of high-quality proprietary data and 74,000 hours of open-source data. In terms of performance, Dolphin's word error rate (WER) is significantly lower than that of Whisper models of the same size. For example, the average WER of the base version is reduced by 63.1%, and that of the small version is reduced by 68.2%. It adopts a CTC-Attention architecture, combines an E-Branchformer encoder and a Transformer decoder, and accelerates computation through a 4x downsampling layer while retaining key speech information.
Main features of Dolphin
- Multilingual and dialect recognition: Dolphin supports speech recognition for 40 Eastern languages, covering a wide range and meeting speech recognition needs in different language environments.
- High-precision speech-to-text: Dolphin can efficiently and accurately convert speech signals into text content. It can maintain high accuracy in speech recognition tasks across multiple languages and dialects, effectively reducing errors and misunderstandings in the speech-to-text process.
- Custom language and region settings: Users can flexibly specify the language and region for speech recognition according to actual needs. Dolphin uses a two-level language tag system. The first tag specifies the language (such as <zh> for Chinese), and the second tag specifies the region (such as <CN> for China). This enables the model to accurately capture the differences between different dialects and accents within the same language, as well as the similarities between different languages within the same region, improving the model's recognition performance in specific language and regional environments and enhancing its generalization ability.
- Open-source support and secondary development: Dolphin's base and small version models and inference code are fully open source, providing great convenience for developers. The open-source nature enables developers to carry out secondary development and optimization of the model according to their own needs, such as fine-tuning the model for specific application scenarios and extending the model's functions, meeting personalized needs and promoting the application and innovation of speech recognition technology in more fields.
- Convenient and quick usage: Dolphin provides multiple usage methods, including command-line invocation and Python interface invocation. Users can quickly start speech recognition tasks through simple command-line instructions, and can also flexibly call the model in a Python environment for speech processing and analysis, making it convenient to integrate with other applications or systems and improving development efficiency and application flexibility.
Technical principles of Dolphin
- CTC-Attention architecture: Dolphin adopts the CTC-Attention architecture, combining the sequence modeling capability of CTC (Connectionist Temporal Classification) with the context-capturing capability of the attention mechanism. It can effectively capture subtle changes in speech information when processing complex audio input, maintain efficient processing speed, and improve the model's recognition accuracy and efficiency.
- E-Branchformer encoder: The E-Branchformer encoder adopts a parallel branch structure, which can more effectively capture the local and global dependencies of the input speech signal. It provides the model with richer feature representations, enabling the model to better understand complex patterns in speech signals and improve recognition accuracy.
- Transformer decoder: Dolphin's decoder part adopts the Transformer architecture. Transformer performs excellently in sequence-to-sequence tasks and can generate high-quality text output. Through the self-attention mechanism, it can effectively capture long-distance dependencies in the text, ensuring the quality and coherence of the final output text.
- 4x downsampling layer: To further improve training efficiency and performance, Dolphin introduces a 4x downsampling layer. It can reduce the sequence length of input features, accelerate the computation process, preserve key speech information, and ensure that the model's recognition performance is not affected.
- Two-level language tag system: Dolphin introduces an innovative two-level language tag system. The first tag specifies the language (for example, <zh> indicates Chinese), and the second tag specifies the region (for example, <CN> indicates China). This enables the model to capture the differences between different dialects and accents within the same language, as well as the similarities between different languages within the same region, improving the model's ability to distinguish closely related dialects and enhancing generalization ability.
Dolphin's project address
- Github repository：https://github.com/DataoceanAI/Dolphin
- HuggingFace model library：https://huggingface.co/DataoceanAI
- arXiv technical paper：https://arxiv.org/pdf/2503.20212
Dolphin's application scenarios
- Meeting minutes: Automatically convert speech content in meetings into text records, improving work efficiency.
- Voice input method: On smart devices, users can input text through voice, reducing the trouble of manual input and improving input speed and convenience.
- Smart voice assistant: Supports multiple languages and dialects, can better understand users' voice commands, provide more accurate feedback, and reduce misunderstandings caused by dialects or accents.
- Smart home: Users can control smart home devices such as lights and air conditioners through voice, improving the intelligence and convenience of the home.
- News media: Quickly convert speech content such as news interviews and podcasts into text, facilitating editing and publishing.
- Language learning: Supports recognition of multiple languages and dialects, and can help students practice pronunciation and language expression.