Audio-Visual Flamingo - An Open-Source Audio-Visual Language Model from NVIDIA and Others

What is Audio-Visual Flamingo

Audio-Visual Flamingo (AV-Flamingo for short) is an open-source audio-visual large language model jointly launched by NVIDIA and the University of Maryland, designed specifically for joint understanding and reasoning over long videos and complex real-world audio-visual content. The model can process both visual and auditory inputs simultaneously and also supports speech interaction output. Across more than 15 audio-visual, omni-modal, and visual benchmarks, it surpasses many larger open-source and closed-source models with a 7B parameter scale.

Main Features of Audio-Visual Flamingo

• Long Video Audio-Visual Joint Understanding: Supports video inputs up to 15 minutes long, simultaneously parsing visuals, speech, environmental sounds, and music.
• Cross-Modal Reasoning: Answers questions that require combining both visual and auditory information (such as "what image corresponds to a certain sound in the video").
• Temporal Grounding Reasoning: Through the TAVIT framework, reasoning steps are explicitly anchored to video timestamps, improving temporal alignment and interpretability.
• Speech Dialogue Output: Integrates streaming TTS, supporting real-time speech replies based on audio-visual content.
• Multi-Skill Coverage: Covers 13 categories of capabilities, including relational reasoning, emotion change tracking, causal reasoning, spatiotemporal perception, hallucination detection, event counting, and more.

Technical Principles of Audio-Visual Flamingo

• Architecture: Uses a SigLip visual encoder + AF-Whisper audio encoder to separately extract image/video frame features and audio features; a 2-layer MLP projector maps both into the LLM embedding space.
• Cross-Modal Temporal Alignment: Interleaves visual and audio tokens in time windows and introduces Constrained Rotary Time Embedding (CRTE), enabling the model to perceive the absolute timing and relative order of multimodal events.
• Three-Stage Training Curriculum: (1) Pretraining: uses AV-Skills short-context data; (2) Mid-training: uses AV-Skills long-context data; (3) Post-training: uses chain-of-thought data, gradually transitioning from short-range perception to long-range multi-event reasoning.
• Dataset: Audio-Visual-Skills contains approximately 7 million training instances, covering 240,000 hours of video, with a special emphasis on temporal, compositional, and cross-modal reasoning.
• Base Model: Based on Qwen2.5-7B, it adopts hybrid sequence parallelism and fully sharded data parallelism strategies to handle long contexts.

Follow WeChat and reply "Open Source" to join theAI Open Source Project Communication Group

How to Use Audio-Visual Flamingo

• Obtain the Model: Visit the AV-Flamingo GitHub repository or Hugging Face page, and download the open-source model weights, inference code, and configuration files.
• Environment Setup: Configure a Python environment and install the project dependencies, ensuring that GPU memory is sufficient to load the 7B parameter model and its multimodal encoders.
• Prepare Input: Use images, videos, or audio files to be analyzed as input, and also write specific text questions or instructions.
• Visual Encoding: Visual input is processed by the SigLip encoder to extract multi-scale spatial features, which are compressed by the Dynamic S² module into a compact sequence of visual tokens.
• Audio Encoding: After resampling, the audio input is processed by the AF-Whisper encoder to extract features using a 30-second sliding window, supporting audio streams of arbitrary length.
• Cross-Modal Alignment: The model automatically maps audio-visual features into the LLM embedding space through a projection layer, interleaves them by time window, and adds rotary temporal embeddings to preserve absolute temporal order.
• Inference Generation: The processed multimodal tokens are concatenated with the text prompt and fed into Qwen2.5-7B for autoregressive inference to generate textual responses.

Core Advantages of Audio-Visual Flamingo

• Long-Video Specialization: Optimized for long videos of over 1 minute and even up to 15 minutes, addressing the performance degradation of existing models on long videos.
• True Joint Reasoning: Both the dataset and training objectives are designed for cross-modal tasks, rather than simply combining unimodal capabilities.
• Temporal Interpretability: TAVIT enables the model, when answering questions about long videos, to explicitly cite specific time intervals such as "from the 22nd second to the 45th second" as the basis for reasoning.
• Fully Open Source: Compared with closed-source solutions such as GPT-4o and Gemini, AV-Flamingo is fully open at the model, data, and code levels.

Audio-Visual Flamingo Project Links

• Project Website：https://avflamingo.pages.dev/
• GitHub Repository：https://github.com/NVIDIA/audio-flamingo
• HuggingFace Model Hub：https://huggingface.co/nvidia/nemotron-labs-audio-visual-flamingo-hf
• arXiv Technical Paper：https://arxiv.org/pdf/2607.16107

Comparison of Audio-Visual Flamingo with Similar Competitors

Application Scenarios of Audio-Visual Flamingo

• Film and Television Content Analysis: Automatically generates detailed timestamped commentary for feature films/TV series, locating key plot points and sound effects.
• Education and Online Courses: Enables cross-modal question answering over long lecture videos, allowing students to ask "at what minute did the teacher mention a certain formula?"
• Sports and Event Review: Combining visuals with live commentary, analyze tactical changes, goal moments, and commentator viewpoints.
• Podcast and Interview Processing: Generate structured summaries for audio-only or audio-video podcasts, and support precise timestamp-based retrieval.
• Security and Surveillance Review: In long surveillance footage, jointly locate anomalous events through audio (such as breaking glass) + visuals.