MOSS-VL-Realtime - OpenMOSS's Open-Source Vision-Language Model

What Is MOSS-VL-Realtime

MOSS-VL-Realtime is an 11B-parameter streaming vision-language model open-sourced by OpenMOSS, designed specifically for real-time video understanding. The model supports answering while watching, instant correction, and proactive silence, upgrading video understanding from watching recordings to watching live streams. Through a dual-channel cross-attention architecture and absolute timestamp encoding, the model achieves parallel perception and generation, keeping AI temporally synchronized with the physical world and reaching open-source SOTA on multiple streaming benchmarks.

MOSS-VL-Realtime's Main Features

Real-Time Video Understanding: Continuously receives video streams, generating answers while watching without waiting for the video to end.
Instant Q&A Response: Users can insert questions at any time, and the model responds instantly based on the current frame.
Proactive Silence Mechanism: When information is insufficient or there are no key events, the model autonomously remains silent and waits.
Dynamic Answer Correction: When the scene changes, the model adjusts and corrects previous answers in real time.
Spatiotemporal Localization Awareness: Through absolute timestamp encoding, it precisely perceives the physical time stream and spatial positions.
Long-Context Processing: Supports continuous understanding and interaction with long video streams.

MOSS-VL-Realtime's Technical Principles

Dual-Channel Cross-Attention Architecture: Visual features enter through a side channel and are decoupled from text generation, allowing perception and generation to run in parallel without blocking each other.
Decoupled Cross-Attention: Separates the cross-attention computation of video feature extraction and text generation, reducing the end-to-end latency of high-frequency frames.
Absolute Timestamp Encoding: Embeds absolute time information into each frame, enabling the model to perceive the physical time stream and eliminating dependence on a fixed frame rate.
XRoPE Cross-Dimensional Rotary Position Encoding: Uniformly maps the spatial dimensions (h, w) and temporal dimension (t) of video features to construct a unified spatiotemporal representation.
Streaming SFT Training Paradigm: Restructures multimodal corpora, converting dense captions into Q&A data with real-time revisions, training the model to predict when to speak and when to stay silent.

Follow WeChat and reply "open source" to join theAI Open-Source Project Exchange Group

How to Use MOSS-VL-Realtime

Obtain Model Resources: Go to the OpenMOSS official websitehttps://openmoss.ai/MOSS-VL/浏览演示案例并下载开源权重与代码。
Prepare the Hardware Environment: A single H200 GPU is recommended, supporting real-time inference and low-latency response for 256-frame video sequences.
Deploy Model Weights: Load the 11B-parameter model into the local environment, completing dependency installation and weight initialization configuration.
Connect to a Video Stream: Connect a real-time camera or import a video stream input to ensure frame data can be continuously transmitted to the model.
Start Real-Time Interaction: Enter text questions at any time during video playback, and the model will instantly generate or revise answers based on the current frame.

Core advantages of MOSS-VL-Realtime

Streaming open-source SOTA: Achieves open-source state-of-the-art performance on multiple streaming video understanding benchmarks, with industry-leading real-time interaction capabilities.
Ultra-low latency and high throughput: On a single H200 GPU, first token generation time is accelerated by about 5x, and decoding throughput is increased by 2.7x.
Offline capability without degradation: V*Bench 89.0, BLINK 78.0; long video and document chart understanding still aligns with top open-source baselines.
Human-like interaction rhythm: Autonomously determines when to speak, naturally supporting interruption, instant correction, proactive silence, and other natural conversational behaviors.
Unified streaming representation: Video frames, user questions, and model responses are interwoven into a single token sequence, enabling seamless real-time interaction.

MOSS-VL-Realtime project address

Project website：https://openmoss.ai/MOSS-VL/
HuggingFace model repository：https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime

Comparison of MOSS-VL-Realtime with similar competitors

Dimension	MOSS-VL-Realtime	Qwen2.5-VL-7B
Architecture	Cross-attention dual-channel, perception and generation decoupled and parallel	Decoder-only, visual tokens embedded in autoregressive sequence
Interaction mode	Streaming real-time, watch and speak simultaneously, instant correction	Offline batch processing, requires watching the entire video before answering uniformly
First token latency	About 5x speedup, millisecond-level response	Baseline level, blocked by visual processing
Decoding throughput	2.7x improvement, smooth generation at high frame rates	Baseline level
Proactive behavior	Supports proactive silence, instant correction, arbitrary interruption	No real-time behavior control capability
Temporal awareness	Absolute timestamp encoding, perceives the physical flow of time	Relative position encoding, depends on fixed frame rate
Model scale	11B open-source weights	7B open-source weights

Application scenarios of MOSS-VL-Realtime

Real-time monitoring alerts: In street monitoring scenarios, it automatically stays silent when nothing happens, and instantly triggers an alert when a target person falls or an abnormal intrusion occurs.
Live event commentary: Captures key actions such as passes, shots, and goals in real time during football and other sports events, with synchronized professional commentary.
Speech slide companion reading: Follows PPT page-turning progress in real time, delivering impromptu and coherent narration for each page from start to finish.
Experiment process recording: Watch plant growth time-lapse photography, accurately capture key changes and accurately report the specific number of days on which they occur.
Real-time teaching tutoring: Watch the handwritten problem-solving process, transcribe each line of formulas as it is written and simultaneously explain the reasoning logic.