JoyAI-VL-Interaction - JD.com's Open-Source Real-Time Video Vision-Language Interaction Model

What is JoyAI-VL-Interaction

JoyAI-VL-Interaction is a real-time video vision-language interaction model open-sourced by JD.com's Joy Future Academy, and is the world's first full-stack open-source interaction model system. JoyAI-VL-Interaction shifts large models from question-and-answer to watching-and-speaking, enabling continuous observation of video streams, autonomous judgment of when to speak, real-time response to scene changes, and support for delegating complex tasks to backend Agents. In 58 human blind evaluations, it achieved a 77.6% win rate against Doubao Video Call Assistant and an 87.9% win rate against Gemini.

Main Features of JoyAI-VL-Interaction

• Proactive Visual Response: Continuously observes camera/live streams/surveillance streams, autonomously judges when to speak and when to stay silent, without requiring users to ask round by round.
• Real-Time Streaming Interaction: Provides immediate responses to ongoing video streams, giving second-level feedback when the scene changes, rather than summarizing the complete video after the fact.
• Agent Delegation: When encountering tasks such as code generation, tool calling, or complex reasoning, automatically hands them off to a backend large model or Agent while the frontend continues observing.
• Multimodal Input and Output: Supports voice input and output, a visual interface, and long-term memory; ASR/TTS/interface can all be replaced as needed.
• Long-Term Memory: Possesses minute-level visual memory, capable of recalling visual details from several minutes earlier and answering accurately.

Technical Principles of JoyAI-VL-Interaction

• Vision-First Interaction Decision-Making: The core of the model is a decision automatically executed every second—speak, stay silent, or delegate. The model is built on the JoyAI-VL-8B vision-language instruction model, treats voice as pluggable input and output, and its sole task is to observe the scene and determine the timing for action.
• Predictive Video Codec: Adopts the AdaCodec predictive video codec, consuming only a small number of tokens for predictable frames and preserving full details when the scene actually changes, allowing the token budget to grow slowly over time rather than expanding frame by frame, thereby supporting long-duration real-time stream processing.
• Temporally Aligned Behavior Learning: The model is trained on over 4 million second-by-second annotated temporal interaction clips, with each data point precisely labeling when to speak, stay silent, or delegate, supplemented by reinforcement learning fine-tuning, allowing behavior to be learned from data.
• Pluggable System Architecture: Builds a complete deployment system around the core model, including streaming ASR/TTS, a long-term memory module, a visual UI, and backend model bridging. All components can be replaced independently, and the whole system runs on standard vLLM infrastructure, with day-0 native support from vLLM-Omni.

Follow WeChat and reply "open source" to join theAI Open-Source Project Communication Group

How to Use JoyAI-VL-Interaction

• Clone the Repository: Visit the GitHub open-source repository jd-opensource/JoyAI-VL-Interaction to obtain the complete code and deployment system.
• Obtain the Model: Download the jdopensource/JoyAI-VL-Interaction-Preview model weights from Hugging Face.
• Prepare the Environment: Deploy based on standard vLLM infrastructure, supporting various video inputs such as cameras, RTSP surveillance streams, and live streams.
• Customize Components: Replace ASR, TTS, voice services, Agent, API, or frontend interface as needed, and connect to your own business systems.
• Running experience: After startup, point it at a camera or live stream, and the model immediately enters a state of real-time observation and interaction.

Core advantages of JoyAI-VL-Interaction

• Fully open source: The 8B model, training recipe, 4 million temporal data samples, and the complete deployable system are all open, reproducible, and extensible.
• Real-time presence: It can continuously observe in a live stream for hours, with response latency below 1 second, truly "present" rather than awakened by polling.
• Vision-triggered proactivity: It autonomously decides when to speak based on the visual content, enabling zero-latency alerts in scenarios such as surveillance warnings and real-time translation.
• Frontend-backend separation: The frontend model keeps visual observation uninterrupted, while the backend handles complex tasks, and results are naturally fed back into the conversation without interrupting the interaction flow.
• Lightweight and deployable: With a compact 8B parameter scale, it can run on standard infrastructure, lowering the deployment barrier for real-time AI assistants.

Project address of JoyAI-VL-Interaction

• Project official website：https://joyai-vl-video-future-academy-jd.github.io/JoyAI-VL-Interaction/
• GitHub repository：https://github.com/jd-opensource/JoyAI-VL-Interaction/
• HuggingFace model library：https://huggingface.co/datasets/jdopensource/JoyAI-VL-Interaction
• Technical paper：https://echovideo.jd.cn/JoyAI-VL-Interaction/JoyAI-VL-Interaction-Reportv1.pdf

Comparison of JoyAI-VL-Interaction with similar competitors

Application scenarios of JoyAI-VL-Interaction

• Security surveillance: Observes surveillance streams in real time and immediately gives voice alerts for abnormal events (such as falls or intrusions), without requiring manual screen monitoring.
• Elderly/child care: Continuously observes home scenes and proactively alerts when dangerous behavior is detected (approaching the stove, going out alone).
• Live streaming tours/e-commerce shopping guides: Provides real-time commentary on live video, automatically introduces product details, or proactively gives styling suggestions based on the user's outfit.
• Real-time translation: When watching foreign-language videos or communicating face-to-face, continuously recognize subtitles/dialogue and provide real-time spoken translation.
• Operation guidance: Guide users in operating an app or device, providing step-by-step prompts as the screen changes, rather than only describing static screenshots.
• AI glasses/accessibility assistance: As the core visual brain of AI glasses, describe the surrounding environment in real time for visually impaired people and proactively alert them to obstacles.