What is URO-Bench
URO-Bench is a comprehensive benchmarking tool for end-to-end spoken dialogue models (SDMs). It covers multi-dimensional tasks such as multilingualism, multi-turn dialogue, and paralinguistic information, comprehensively evaluating the performance of spoken dialogue models. The benchmark includes a Basic Track and a Pro Track. The Basic Track has 16 datasets, involving open-ended question answering, factual question answering, etc.; the Pro Track includes 20 datasets, covering more complex tasks such as code-switching question answering, speech emotion generation, and multilingual question answering.
Main features of URO-Bench
- Multilingual support: Covers multiple languages, including English and Chinese, and supports cross-lingual dialogue tasks.
- Multi-turn dialogue evaluation: Includes multi-turn dialogue tasks to evaluate the model's performance in continuous conversations.
- Paralinguistic information evaluation: Involves tasks related to paralinguistic information such as speech emotion understanding and speech style generation, staying closer to real speech interaction scenarios.
- Basic Track: Includes 16 datasets, covering various task types such as open-ended question answering, ethics summarization, factual question answering, and math word problems.
- Pro Track: Includes 20 datasets, involving more advanced tasks such as code-switching question answering, speech emotion generation, multilingual question answering, and audio understanding.
- Four-step evaluation process: Users can quickly obtain the model's results on all test sets through simple steps such as modifying the inference code, configuring scripts, and running the automatic evaluation pipeline. Sample code and scripts are provided to help users get started quickly and lower the barrier to use.
- Multi-metric evaluation: Comprehensively evaluates the model's capabilities in speech understanding, reasoning, and spoken dialogue through multiple metrics (such as UTMOS, ASR-WER, emotion understanding accuracy, etc.).
- Generalizability: Supports a variety of end-to-end spoken dialogue models, and users can integrate their own models into URO-Bench for evaluation.
- Reference models: Provides evaluation results of some pretrained models (such as Whisper + GPT-4o, GLM-4-Voice, etc.) as references.
Technical principles of URO-Bench
- Text-to-speech (TTS): Uses advanced TTS systems (such as F5-TTS and CosyVoice) to convert text data into speech data.
- Automatic speech recognition (ASR): Uses ASR systems such as Whisper-large-v3 to transcribe speech data into text for evaluation.
- Emotion recognition: Uses models such as emotion2vec to evaluate emotional information in speech.
- Multilingual processing: Supports input and output in multiple languages, and evaluates the model's cross-lingual capabilities.
Project address of URO-Bench
- GitHub repository：https://github.com/Ruiqi-Yan/URO-Bench
- HuggingFace model library：https://huggingface.co/datasets/Honggao/URO-Bench
- arXiv technical paper：https://arxiv.org/pdf/2502.17810
Application scenarios of URO-Bench
- Smart home control: Users can control smart devices at home through voice commands, such as lights, temperature adjustment, and appliance switches. URO-Bench can evaluate the performance of voice assistants in understanding user commands and generating natural speech feedback.
- Personal assistant: Voice assistants can help users schedule appointments, remind them of important events, and query information. URO-Bench can test the coherence and accuracy of voice assistants in multi-turn dialogues.
- Language Learning: Speech dialogue models can serve as language learning tools to help users practice speaking and listening. URO-Bench can evaluate the model's performance in multilingual dialogue and emotional expression, providing language learners with a more natural interactive experience.
- Medical Consultation: Speech dialogue models can provide users with preliminary medical consultation and answer common health questions. URO-Bench can evaluate the model's performance in understanding and generating professional medical information.
- Voice Games: Speech dialogue models can be used to develop voice-interactive games, providing a more immersive experience. URO-Bench can evaluate the model's performance in multi-turn dialogue and emotional generation, providing a reference for game development.