What is OmniAudio-2.6B
OmniAudio-2.6B is an audio language model by Nexa AI, designed for edge deployment to achieve fast and efficient audio-text processing. OmniAudio-2.6B is a multimodal model with 2.6 billion parameters that fuses Gemma-2-2b, Whisper Turbo, and a custom projection module, optimizing the integration of automatic speech recognition and language models to reduce latency and resource consumption. On the 2024 Mac Mini M4 Pro, OmniAudio-2.6B demonstrates decoding speeds 5.5 to 10.3 times faster than Qwen2-Audio-7B-Instruct. OmniAudio-2.6B is suitable for various scenarios, including voice question answering, dialogue, and creative content generation. It runs locally on devices via the Nexa SDK, providing users with a powerful edge AI solution.
Main Features of OmniAudio-2.6B
- Speech Recognition and Transcription: Converts speech input into text, suitable for scenarios such as meeting minutes and voice notes.
- Voice Question Answering: Users ask the model questions via voice, and the model understands the question and provides a text answer.
- Voice Dialogue: The model can participate in voice conversations, understand voice input, and generate corresponding text replies.
- Creative Content Generation: Users can ask the model to generate creative content based on voice input, such as poems and stories.
- Recording Summarization: The model can understand and summarize long voice recordings, providing summaries of key information.
Technical Principles of OmniAudio-2.6B
- Integrated Architecture: Integrates Gemma-2-2b, Whisper Turbo, and a custom projection module to reduce the latency and resource consumption caused by the traditional serial connection of ASR and LLM models.
- Sparsity Utilization: Based on the sparsity of the language model embedding space, the projection module maps Whisper's audio tokens to a sequence aligned with Gemma's text embeddings, achieving effective fusion of audio and text.
- Three-Stage Training Process：
- Pre-training: Uses the MLS English 10k transcription dataset, introducing special tokens to distinguish transcription and completion tasks.
- Supervised Fine-Tuning (SFT): Creates a synthetic dataset based on the transcription dataset, used for instruction tuning to enable the model to understand and process conversational audio input.
- Direct Preference Optimization (DPO): Evaluates model outputs based on the GPT-4o API, identifies and optimizes erroneous responses, and improves model accuracy.
- Efficient Inference Engine: Nexa SDK is a C++ inference engine based on GGML, designed for deploying audio language models on edge devices, enabling efficient audio language model inference.
- Quantization and Optimization: The model supports FP16 and Q4_K_M quantized versions, reducing memory and storage requirements, adapting to resource-constrained edge devices.
OmniAudio-2.6B Project Address
- Project Website：nexa.ai/blogs/omniaudio-2.6b
- HuggingFace Model Repository：https://huggingface.co/NexaAIDev/OmniAudio-2.6B
- Online Experience Demo：https://huggingface.co/spaces/NexaAIDev/omni-audio-demo
Application Scenarios of OmniAudio-2.6B
- Smart Assistants and Virtual Assistants: As the core of voice interaction in smartphones and smart home devices, providing fast-response speech recognition and natural language understanding functions.
- In-Vehicle Systems: Integrated into cars to provide voice control, navigation, entertainment system operation, and other functions, improving driving safety and convenience.
- Meeting Minutes and Transcription: Automatically records and transcribes meeting content during business meetings, generates meeting summaries, and improves work efficiency.
- Education and learning: Assists language learning, provides speech recognition and feedback, and helps learners improve pronunciation and language ability.
- Healthcare: In medical environments, controls medical devices through voice commands, or provides voice interaction services for patients.