What is Lyra
Lyra is an efficient multimodal large language model (MLLM) launched by The Chinese University of Hong Kong, SmartMore, and The Hong Kong University of Science and Technology, focusing on enhancing the interaction capabilities of speech, vision, and language modalities. Lyra is based on open-source large models, multimodal LoRA modules, and a latent multimodal regularizer, reducing training costs and data requirements. Lyra constructs large-scale multimodal datasets, including long speech samples, handles complex long speech inputs, and achieves powerful omni-modal cognitive capabilities. In various modality understanding and reasoning tasks, Lyra achieves state-of-the-art performance while being more efficient in the use of computational resources and training data.
Lyra's Main Features
- Multimodal Understanding and Reasoning: Lyra can understand and process data from multiple modalities such as images, videos, audio, and text, and perform complex understanding and reasoning tasks.
- Speech-Centric Capabilities: The model particularly strengthens speech understanding, including recognition and processing of long speech, and performs excellently in speech interaction.
- Efficient Processing: Lyra is more efficient during training and inference, using less data and computational resources, making it suitable for real-time and long-context multimodal applications.
- Streaming Generation: Supports simultaneous generation of text and speech output, enabling real-time responses in dialogue and interaction.
- Cross-Modal Interaction: Based on a latent multimodal regularizer and extractor, it strengthens information interaction between different modalities and improves model performance.
Lyra's Technical Principles
- Multimodal LoRA (Low-Rank Adaptation): Based on LoRA technology adapted for multimodal input, the model retains its original visual capabilities while developing capabilities in the speech modality, reducing training data requirements.
- Latent Cross-Modal Regularizer: Based on the Dynamic Time Warping (DTW) algorithm, it aligns speech tokens with corresponding text tokens, ensuring that speech modality input remains semantically consistent with the text modality.
- Latent Multimodal Extractor: Based on evaluating the relevance of tokens from different modalities to the text query, it dynamically selects and retains the tokens most relevant to the task, improving training and inference efficiency.
- Long Speech Capability Integration: Constructs a dedicated long speech SFT dataset, processes long speech tokens based on compression technology, and enables the model to handle audio inputs lasting several hours.
- Streaming Text-Speech Generation: Integrates a streaming generation mechanism, supporting the model to output corresponding speech while generating text, achieving a seamless multimodal interaction experience.
- Dataset Construction: To train and optimize Lyra, researchers constructed a high-quality dataset containing more than 1.5 million multimodal samples and more than 12,000 long speech samples, with data covering rich scenarios and domains.
Lyra's Project Address
- Project Official Website：lyra-omni
- GitHub Repository：https://github.com/dvlab-research/Lyra
- HuggingFace Model Library：https://huggingface.co/collections/zszhong/lyra-data
- arXiv Technical Paper：https://arxiv.org/pdf/2412.09501
Lyra's Application Scenarios
- Intelligent Assistant: As an intelligent assistant, it understands and responds to users' voice commands, providing services such as information queries, schedule management, and reminder settings.
- Customer Service: In the customer service field, based on speech and text interaction, it handles issues such as customer inquiries, complaints, and technical support.
- Education and Training: As an educational assistance tool, it provides voice explanations, understanding of course content and question answering, as well as pronunciation and listening training in language learning.
- Health and Medical Care: In the medical field, it helps patients consult about health issues through voice, or serves as an auxiliary tool for doctors to understand and summarize patients' medical records.
- Content Moderation: Analyzes images, videos, and text content to perform content moderation, identifying and filtering inappropriate content.