JoyAI-Video-Edit - JD.com's Open-Source Real-Time Streaming Video Editing Model

What is JoyAI-Video-Edit

JoyAI-Video-Edit is a real-time streaming video editing model independently developed and open-sourced by JD.com. Based on a 16B-parameter autoregressive diffusion architecture, it achieves 30FPS inference speed at 720P resolution and supports stable streaming editing of arbitrary duration. Users can modify characters, scenes, styles, and objects in real time during video playback through natural language instructions, without waiting for the complete video to be generated. The model surpasses all streaming editing methods in the OpenVE-Bench evaluation, leading comprehensively across all metrics, and can also provide a large-scale data synthesis path for embodied intelligence.

Main Features of JoyAI-Video-Edit

• Real-Time Streaming Editing: Edits instantly when video frames arrive, without needing to know the complete sequence in advance.
• Open-Ended Instruction Control: Supports global style transfer, local object addition, deletion, and modification, background replacement, action adjustment, and reference-image-guided editing.
• Arbitrary-Duration Processing: Breaks through second-level or minute-level limits, enabling stable editing continuously as the video plays.
• 720P High-Throughput Deployment: The end-to-end pipeline achieves 30.19 FPS at 720×1280 resolution.
• Multi-Dimensional Editing Capabilities: Covers tasks such as character outfit changes, scene transformation, style transfer, object replacement, and subtitle editing.

Technical Principles of JoyAI-Video-Edit

• MLLM Conditional Encoder: Uses a multimodal large language model to encode natural language instructions into editing conditions, enabling open-ended semantic control.
• Causal Video VAE: Employs a variational autoencoder with a causal temporal structure to compress and reconstruct video frames, ensuring temporal consistency in streaming scenarios.
• 16B-Parameter MMDiT Backbone: Uses a 16B multimodal diffusion Transformer as the core, fusing text and visual features for autoregressive diffusion generation.
• Autoregressive Distribution Matching Distillation: Accelerates inference through a distillation strategy that aligns autoregressive distributions, significantly reducing the distribution gap between training and inference.
• Long-Horizon Optimization and Bounded KV Inference: Introduces long-horizon stability optimization and a bounded KV cache mechanism to suppress cumulative temporal drift and maintain high throughput.

Follow WeChat and reply "Open Source" to join theAI Open-Source Project Communication Group

How to Use JoyAI-Video-Edit

• Environment Preparation: Create a Conda environmentpython=3.10and installrequirements.txtdependencies.
• Download Weights: Obtain the model files from Hugging Face and place the checkpoints according to the specified directory structure.
• Start the Service: Enter thedeploydirectory and runbash run_server.shto start the local server.
• Access the Interface: Openhttp://localhost:8080in a browser to enter the interactive editing interface.
• Real-Time Editing: Upload a video or connect a camera stream, and input natural language instructions to edit while playing.

Core Advantages of JoyAI-Video-Edit

• Real-Time Streaming Editing: Processes video frames as they arrive, without waiting for the complete sequence, delivering a true "edit while playing" interactive experience.
• Both Speed and Quality Are Excellent: End-to-end inference reaches 30.19 FPS at 720P resolution, surpassing all streaming and offline editing methods in the OpenVE-Bench evaluation.
• Stable output for arbitrary durations: Breaks through the limitation of traditional models that only support second-level or minute-level clips, enabling continuous drift-free stable editing as the video plays.
• Open-ended semantic control: Based on an MLLM conditional encoder, it supports multi-dimensional editing such as global style, local objects, background replacement, and reference image guidance driven by natural language instructions.
• Efficient inference architecture: A 16B-parameter autoregressive diffusion Transformer combined with distribution matching distillation and bounded KV cache significantly reduces the training-inference discrepancy and suppresses temporal cumulative errors.

Project address of JoyAI-Video-Edit

• GitHub repository：https://github.com/jd-opensource/JoyAI-Video-Edit
• HuggingFace model library：https://huggingface.co/jdopensource/JoyAI-Video-Edit
• arXiv technical paper：https://arxiv.org/pdf/2608.03974

Comparison of JoyAI-Video-Edit with similar competitors

Application scenarios of JoyAI-Video-Edit

• Short video creation: Creators can replace characters' clothing and adjust scene styles in real time during video playback, improving content production efficiency.
• Live streaming real-time effects: Streamers can change backgrounds or add virtual elements while broadcasting, achieving interactive live-streaming visual enhancement.
• Home decoration and interior design: Users can replace furniture, wall materials, and lighting effects in real time while browsing room videos, assisting decoration decisions.
• Film and television post-production preview: Directors can quickly try different visual styles and scene replacements during the editing stage, reducing trial-and-error costs.
• Embodied intelligence data synthesis: Converts human hand operation videos into robotic arm operation materials, expanding the scale of robot training data at low cost.