What is CoGenAV
CoGenAV (Contrastive-Generative Audio-Visual Representation Learning) is an advanced multimodal learning model focused on the alignment and fusion of audio and visual signals. It is trained through the dual objectives of contrastive feature alignment and generative text prediction, utilizing synchronized audio, video, and text data to learn to capture temporal correspondences and semantic information. CoGenAV can be trained with only 223 hours of labeled data, demonstrating extremely high data efficiency.
Main Features of CoGenAV
- Audio-Visual Speech Recognition (AVSR): CoGenAV can combine audio and visual information (such as the speaker's mouth movements) to improve the accuracy of speech recognition.
- Visual Speech Recognition (VSR): Speech recognition using only visual information (such as the speaker's mouth movements), without relying on audio signals.
- Speech Processing in Noisy Environments: In high-noise environments, CoGenAV uses visual information to assist audio signals, improving the robustness of speech processing.
- Speech Reconstruction and Enhancement: CoGenAV can be used for speech reconstruction and enhancement tasks, improving speech quality through multimodal information fusion.
- Active Speaker Detection (ASD): Detecting who is currently speaking by analyzing audio and visual signals.
Technical Principles of CoGenAV
- Feature Extraction: In the feature extraction stage, CoGenAV uses ResNet3D CNN to analyze the speaker's lip movements in the video, capturing the dynamic association between sound and mouth shape. It uses a Transformer encoder to extract speech information from audio, precisely aligning audio-visual features.
- Contrastive Synchronization: It adopts the Seq2Seq Contrastive Learning method to enhance the correspondence between audio and video features, introduces a ReLU activation function to filter interfering frames, and improves the model's stability in complex environments.
- Generative Synchronization: With the help of a pre-trained ASR model (such as Whisper), audio-visual features are aligned with their acoustic-text representations, and a lightweight adaptation module (Delta Upsampler + GatedFFN MHA) is designed to effectively improve cross-modal fusion efficiency.
CoGenAV Project Address
- Github Repository：https://github.com/HumanMLLM/CoGenAV
- HuggingFace Model Library：https://huggingface.co/detao/CoGenAV
- arXiv Technical Paper：https://arxiv.org/pdf/2505.03186
Application Scenarios of CoGenAV
- Intelligent Assistants and Robots: CoGenAV's multimodal representation can be integrated into intelligent assistants and robots to better understand and respond to voice commands in complex environments.
- Video Content Analysis: CoGenAV can be used for the analysis and understanding of video content, providing more accurate subtitle generation, content recommendation, and other functions by analyzing audio and visual information in videos.
- Industrial Applications: In industrial environments, CoGenAV can be used for voice-controlled devices, voice monitoring, and other scenarios, improving the robustness and reliability of systems through multimodal information fusion.
- Healthcare: CoGenAV can be used for voice interaction in medical devices, such as intelligent medical assistants and voice-controlled medical devices, enhancing the usability and interactivity of medical devices.