What Is StreamBridge
StreamBridge is an on-device video large language model (Video-LLMs) framework jointly launched by Apple and Fudan University, which can help AI understand live streaming video in real time. The framework supports long-context interaction based on a memory buffer and a round-decay compression strategy, and introduces a lightweight activation model to implement proactive response functionality. The research team has released the Stream-IT dataset containing about 600,000 samples to improve streaming video understanding capabilities. Tests on mainstream offline models such as LLaVA-OV-7B, Qwen2-VL-7B, and Oryx-1.5-7B show that StreamBridge significantly improves the model's capabilities in multi-round real-time understanding and proactive response, demonstrating strong potential in the field of streaming video understanding.
Main Features of StreamBridge
- Multi-Round Real-Time Understanding: Supports long-context multi-round interaction, retaining historical visual and dialogue context when processing the latest video segments.
- Proactive Response: The model can proactively monitor video streams like a human and promptly output feedback without explicit instructions.
- Flexible Integration: Supports seamless integration into existing video large language models without requiring large-scale modifications to the base model.
- Data Support: Provides a large-scale streaming video understanding dataset Stream-IT, containing about 600,000 samples, supporting diverse instruction formats for training and optimizing models.
Technical Principles of StreamBridge
- Memory Buffer: Stores and retrieves embedding information of video frames, supporting multi-round interaction. Each new video frame is independently encoded and appended to the buffer. When a user query is received, the contents in the buffer are flattened into a single input embedding sequence and fed into the language model for response generation.
- Round-Decay Compression Strategy: Before each response generation, if the length of the input embedding exceeds a predefined maximum length, the model starts from the earliest dialogue round and merges visual tokens frame by frame until the total length falls below the maximum length. The merging operation is implemented based on average pooling, ensuring that the most recent visual context is preserved.
- Lightweight Activation Model: The activation model is an independent lightweight multimodal large language model (MLLM) that runs in parallel with the main video large language model. The activation model takes the current frame (and the user query and optionally the previous few frames) as input and outputs a binary signal indicating whether the main model should generate a response. The activation model uses a score head for binary classification (whether to respond), and during training introduces a learnable activation token <ACT> to supervise the activation timing.
- Stream-IT Dataset: Based on filtering semantically related short clips from a large-scale video caption corpus, it generates multi-round question-answer sequences to simulate real real-time user interaction. The dataset contains about 600,000 samples and supports multiple task formats, such as dense video captioning, sequential step recognition, and video-based question answering.
Project Address of StreamBridge
- arXiv Technical Paper：https://arxiv.org/pdf/2505.05467
Application Scenarios of StreamBridge
- Real-Time Video Interaction: Enhances the real-time interactive experience in scenarios such as video conferencing and online education.
- Autonomous Driving Assistance: Processes road condition videos in real time to assist autonomous driving decisions.
- Intelligent Surveillance: Analyzes surveillance video in real time to quickly detect abnormal behavior.
- Robot Vision: Helps robots understand the environment in real time and achieve natural interaction.
- Content creation: Assists video creation and editing, providing real-time content analysis.