What is Chinese-LiPS
Chinese-LiPS is a high-quality Chinese multimodalspeech recognitiondataset jointly created by BAAI and Nankai University, containing 100 hours of speech, video, and manual transcription text, innovatively integrating lip-reading video and speaker slides. The slides are carefully designed by domain experts, ensuring the high quality and richness of the visual images. By combining lip-reading and slide information, the dataset improves speech recognition performance. Experiments show that lip-reading information and slide information can improve ASR performance by about 8% and 25% respectively, and the combination of the two can improve it by about 35%. It targets complex contexts such as Chinese explanations, science popularization, teaching, and knowledge dissemination.
Main Features of Chinese-LiPS
- Improve Speech Recognition Performance: The dataset significantly improves the performance of speech recognition systems by integrating lip-reading information and slide semantic information. Experimental results show that lip-reading information can reduce the character error rate by about 8%, slide information by about 25%, and the combination of the two by about 35%.
- Reduce Error Types: Lip-reading information plays an important role in reducing deletion errors, capturing pronunciation-related details and effectively supplementing parts that are easily missing in speech recognition, such as filler words and speech fragments that are not fully expressed due to hesitation. Slide information significantly reduces substitution errors. Its rich semantic and contextual information provides key recognition clues for the model when recognizing words with specific domain attributes such as technical terms and place names.
- Provide High-Quality Multimodal Data: As a high-quality multimodal Chinese speech recognition dataset, it contains 100 hours of speech, video, and corresponding manual transcription, covering lip-reading video and speaker slides, enabling a more comprehensive exploration of audio-visual speech recognition tasks.
Technical Principles of Chinese-LiPS
- Multimodal Data Fusion: The dataset fuses speech, lip-reading information, text extracted from slides through OCR technology, and semantic information obtained from image and graphic content. The combination of multimodal information provides richer context and clues for speech recognition models, significantly improving recognition accuracy and robustness.
- Role of Lip-Reading Information: Lip-reading can capture pronunciation-related details, such as filler words and speech fragments that are not fully expressed due to hesitation, which are easily missing in speech recognition. With lip-reading information, these can be effectively supplemented, reducing deletion errors.
- Role of Slide Information: Slides contain rich semantic and contextual information. When recognizing words with specific domain attributes such as technical terms and place names, they can provide key recognition clues for the model, greatly reducing substitution errors.
Project Address of Chinese-LiPS
- Project Official Website：https://data.baai.ac.cn/datadetail/Chinese-LiPS
- Github Repository：https://github.com/flageval-baai/Chinese-LiPS
- HuggingFace Model Library：https://huggingface.co/datasets/BAAI/Chinese-LiPS
- arXiv Technical Paper：https://arxiv.org/pdf/2504.15066
Application Scenarios of Chinese-LiPS
- Virtual Teacher: The dataset can help create interactive language learning materials, making virtual teachers' explanations more vivid. By integrating lip-reading information and slide semantic information, virtual teachers can present teaching content more naturally and improve teaching effectiveness.
- Intelligent Tutoring: In intelligent tutoring systems, based on multimodal speech recognition technology, students' questions and needs can be understood more accurately, providing more personalized tutoring solutions.
- Museum and exhibition hall guide: In venues such as museums and exhibition halls, virtual guides can use the multimodal information provided by the dataset to introduce exhibits and exhibition content more vividly and accurately, enhancing the visitor experience.
- Enterprise product introduction: Enterprises can use the dataset to create virtual guides for scenarios such as product introductions and training, improving the efficiency and accuracy of information delivery.