What Is Omnilingual ASR
Omnilingual ASR is an automaticspeech recognitionsystem introduced by Meta AI, supporting over 1,600 languages, including 500 low-resource languages. Omnilingual ASR achieves excellent performance by scaling the wav2vec 2.0 encoder to 7 billion parameters and introducing two decoders, with character error rates below 10% for 78% of languages. The Omnilingual ASR framework is community-driven, and users can extend it to new languages by providing only a small number of samples. At the same time, Meta has open-sourced the Omnilingual ASR Corpus dataset and the Omnilingual wav2vec 2.0, a new self-supervised large-scale multilingual speech representation model, to support the development of global speech technology and promote language equality and cultural exchange.
Main Features of Omnilingual ASR
- Multilingual Speech Transcription: Omnilingual ASR can convert speech in over 1,600 languages into text, including many low-resource languages and languages that have never been transcribed by AI.
- Community Extension Capability: Users can extend the model to new languages by providing a small number of audio and text samples, without requiring large amounts of training data or specialized expertise.
- High Performance and Low Error Rate: In 78% of languages, the character error rate (CER) is below 10%, reaching industry-leading levels.
- Multiple Model Choices: Offers models ranging from a lightweight 300M to a powerful 7B model, suitable for different devices and use cases.
- Open Source and Data Sharing: Open-sources the Omnilingual wav2vec 2.0 model and the Omnilingual ASR Corpus dataset, supporting developers and researchers worldwide in further development and research.
Technical Principles of Omnilingual ASR
- wav2vec 2.0 Extension: Extends the wav2vec 2.0 encoder to 7 billion parameters, enabling the extraction of rich multilingual semantic representations from raw speech data.
- Dual-Decoder Architecture: Uses two decoders, the traditional connectionist temporal classification (CTC) and a Transformer-based decoder, the latter drawing on large language model (LLM) techniques to significantly improve performance on long-tail languages.
- In-Context Learning Capability: Inspired by LLMs, the model can quickly adapt to new languages through a small number of in-context samples, without requiring large-scale training data or complex adjustments.
- Large-Scale Multilingual Dataset: The training corpus integrates public datasets and community-provided speech recordings, covering a large number of low-resource languages and providing the model with a broad linguistic foundation.
Omnilingual ASR Project Address
- Project Website：https://ai.meta.com/blog/omnilingual-asr-advancing-automatic-speech-recognition/
- GitHub Repository：https://github.com/facebookresearch/omnilingual-asr
- HuggingFace Model Library：https://huggingface.co/datasets/facebook/omnilingual-asr-corpus
- Technical Paper：https://ai.meta.com/research/publications/omnilingual-asr-open-source-multilingual-speech-recognition-for-1600-languages/
Application Scenarios of Omnilingual ASR
-
Cross-Language Communication: Helps people from different language backgrounds communicate by voice in real time, breaking down language barriers and promoting international cooperation and cultural exchange.
-
Low-Resource Language Preservation: Provides high-quality speech transcription tools for endangered or low-resource languages, helping with language preservation and inheritance.
-
Education and Learning: Assists teaching in multilingual education, helps students practice pronunciation, or provides instant speech translation for language learners.
-
Voice Assistant Extension: Adds support for more languages to intelligent voice assistants, enabling them to serve a broader user base.
-
Content Creation and Media: Automatically transcribe multilingual video and audio content, improve content creation efficiency, and support multilingual subtitle generation.