What is LLaMA-Omni
LLaMA-Omni is a new model architecture developed by researchers from the Institute of Computing Technology, Chinese Academy of Sciences and the University of Chinese Academy of Sciences, designed to achieve low-latency, high-quality speech interaction with large language models (LLMs). By integrating a pre-trained speech encoder, a speech adaptor, a large language model (LLM), and a real-time speech decoder, it directly generates text and speech responses from speech instructions quickly, omitting the traditional step of first transcribing speech to text, thereby improving response speed. The model is built on the latest LLaMA-3.1-8B-Instruct model and trained using the self-built InstructS2S-200K dataset, generating responses quickly with latency as low as 226 milliseconds. In addition, LLaMA-Omni has high training efficiency and can be trained in less than 3 days on 4 GPUs, laying the foundation for the efficient development of future speech interaction models based on the latest LLMs.
Main Features of LLaMA-Omni
- Low-Latency Speech Recognition: Quickly generates responses from speech instructions, reducing waiting time.
- Direct Speech-to-Text Response: Directly generates text responses without first transcribing speech to text.
- High-Quality Speech Synthesis: While generating text responses, it can also generate corresponding speech output.
- Efficient Training Process: Completes training with relatively few computational resources (such as 4 GPUs) and in a short time (less than 3 days).
- Streaming Speech Decoding: Based on a non-autoregressive streaming Transformer model, it enables real-time speech synthesis.
- Multimodal Interaction: Combines both text and speech modes to provide a more natural and more humanized interaction experience.
Technical Principles of LLaMA-Omni
- Speech Encoder: Uses the pre-trained Whisper-large-v3 model as the speech encoder. It extracts feature representations from the user's speech instructions.
- Speech Adaptor: Maps the output of the speech encoder to the embedding space of the large language model (LLM). It reduces the sequence length through downsampling, enabling the model to process speech input.
- Large Language Model (LLM): Uses Llama-3.1-8B-Instruct as the LLM, with powerful text generation capabilities. It directly generates text responses from speech instructions without an intermediate speech-to-text transcription step.
- Streaming Speech Decoder: Adopts a non-autoregressive (NAR) streaming Transformer architecture. It uses connectionist temporal classification (CTC) to predict the discrete unit sequence corresponding to the speech response.
- Two-Stage Training Strategy: Stage 1: Train the model to directly generate text responses from speech instructions. Stage 2: Train the model to generate speech responses.
- Dataset Construction (InstructS2S-200K)：Contains 200K speech instructions and corresponding text and speech responses. Based onTrain the model to adapt to speech interaction scenarios.
LLaMA-Omni Project Address
- GitHub Repository：https://github.com/ictnlp/LLaMA-Omni
- HuggingFace Model Library：https://huggingface.co/ICTNLP/Llama-3.1-8B-Omni
- arXiv Technical Paper：https://arxiv.org/pdf/2409.06666
Application Scenarios of LLaMA-Omni
- Smart Assistants and Virtual Assistants: Provides voice interaction services on smartphones, smart home devices, and personal computers.
- Customer Service: In call centers and customer support systems, used for speech recognition and response to handle customer inquiries and issues.
- Education and training: Provide voice-interactive learning experiences, including language learning, course instruction, and interactive teaching.
- Medical consultation: In telemedicine and health consultation, use voice interaction to provide medical information and advice.
- Automotive industry: Integrated into in-vehicle systems to provide voice-controlled navigation, entertainment, and communication functions.
- Accessibility and assistive technology: Help visually impaired or mobility-impaired users operate devices and services through voice interaction.