What is OSUM
OSUM (Open Speech Understanding Model) is an open-source speech understanding model launched by the Audio, Speech and Language Processing Research Group at the School of Computer Science, Northwestern Polytechnical University. OSUM combines the Whisper encoder and Qwen2 LLM, and supportsspeech recognition(ASR), speech emotion recognition (SER), speaker gender classification (SGC), and other speech tasks. OSUM is based on the "ASR+X" multi-task training strategy, using modality alignment and target task optimization to achieve efficient and stable training. OSUM is trained on approximately 50,000 hours of diverse speech data, with excellent performance in multiple tasks, and outstanding performance in Chinese ASR and multi-task generalization ability.
Main Features of OSUM
- Speech Recognition: Converts speech to text, supporting multiple languages and dialects.
- Speech Recognition with Timestamps: While recognizing speech content, outputs the start and end times of each word or phrase.
- Speech Event Detection: Identifies specific events in speech (such as laughter, coughing, background noise, etc.).
- Speech Emotion Recognition: Analyzes emotional states in speech (such as happiness, sadness, anger, etc.).
- Speaking Style Recognition: Identifies the speaker's style (such as news broadcasting, customer service conversations, everyday spoken language, etc.).
- Speaker Gender Classification: Determines the speaker's gender (male or female).
- Speaker Age Prediction: Predicts the speaker's age range (such as child, adult, elderly).
- Speech-to-Text Chat: Converts speech input into natural language responses, used in dialogue systems.
Technical Principles of OSUM
- Speech Encoder: Uses the Whisper-Medium model (769M parameters), responsible for encoding speech signals into feature vectors.
- Adaptor: Contains 3 convolutional layers and 4 Transformer layers, used to adapt speech features and language model inputs.
- LLM (Language Model): Based on Qwen2-7B-Instruct, fine-tuned with LoRA (Low-Rank Adaptation), to adapt to multi-task requirements.
- Multi-Task Training Strategy：
- ASR+X Training Paradigm: Simultaneously trains the speech recognition (ASR) task and an additional task (such as SER, SGC, etc.). Based on shared features and optimization objectives, improves the model's generalization ability and stability.
- Natural Language Prompt: Based on providing different natural language prompts to the LLM, guides the model to perform different tasks.
- Data Processing and Training: Approximately 50,000 hours of diverse speech data for multi-task training, with datasets including open-source data and internally processed data. Training is divided into two stages: first, multi-task fine-tuning of the Whisper model, then combining with the LLM for further supervised training.
OSUM Project Address
- GitHub Repository：https://github.com/ASLP-lab/OSUM
- arXiv Technical Paper：https://arxiv.org/pdf/2501.13306v2
- Online Experience Demo：https://huggingface.co/spaces/ASLP-lab/OSUM
OSUM Application Scenarios
- Intelligent Customer Service: Based on speech recognition and emotion analysis, automatically understands customer needs and provides personalized services.
- Smart Home: Recognizes speech commands and background events, optimizes voice interaction experience.
- Educational Tools: Analyzes student speech, provides personalized learning feedback.
- Mental Health Monitoring: Detects emotional changes in speech to assist with mental health assessment.
- Multimedia Content Creation: Automatically generates subtitles and tags to assist with video editing.