JoyAI-Echo - JD's Open-Source Long Audio-Video Generation Framework

What is JoyAI-Echo

JoyAI-Echo is an open-source long audio-video generationframework launched by JD, designed specifically for minute-level multi-shot story generation. Through four major technological innovations—a cross-modal memory bank, memory-driven post-training, Director Agent conversational editing, and lightweight real-time super-resolution—the framework addresses core pain points in long video generation such as character face swapping, timbre mutations, and slow generation. It is the first to achieve up to 5 minutes of highly consistent, interactive, high-definition long video generation, marking JD's entry into the global first tier of long video generation.

Main Features of JoyAI-Echo

• Minute-level multi-shot story generation: Supports generating coherent multi-shot long video sequences from a single prompt JSON, up to 5 minutes in length.
• Cross-modal audio-video joint generation: A single pipeline synchronously outputs video and audio, ensuring audio-video synchronization.
• Paired cross-modal memory bank: Continuously saves and recalls character appearance features and speaker timbres during multi-shot generation, maintaining story-level consistency.
• DMD distillation few-step inference: Achieves approximately 7.5x generation speed improvement through distribution matching distillation technology.
• Director Agent conversational editing: Users can interact with the director assistant in natural language, automatically splitting scripts, characters, scenes, and shots, supporting local revisions without rerunning the entire video.
• Lightweight real-time super-resolution: Supports single-step super-resolution from 736×1280 to 1152×1920 or 1472×2560, maintaining high-definition output under streaming latency constraints.

Technical Principles of JoyAI-Echo

• Cross-modal audio-video memory bank: The core breakthrough of JoyAI-Echo is an built-in paired cross-modal memory bank, which binds visual memory and audio memory for storage through a Slot-paired mechanism. During multi-shot generation, the memory bank continuously saves and recalls characters' facial features, overall appearance, speaker timbres, and audio-video correspondences, so that each new shot is generated conditioned on the identity features of previous shots, maintaining story-level consistency throughout a video up to 5 minutes long, thoroughly solving the problems of character face swapping and timbre mutations.
• Memory-driven post-training and DMD distillation acceleration: The team introduced a memory-driven post-training pipeline combining supervised fine-tuning (SFT), cross-modal RLHF, and Distribution Matching Distillation (DMD) technology. DMD compresses the original multi-step diffusion inference into few-step inference, achieving approximately 7.5x inference acceleration while maintaining generation quality, making streaming generation of minute-level long videos practical from theoretical.
• Director Agent interaction architecture: The framework introduces an intelligent director agent that automatically expands users' natural language intentions into structured scripts, shots, characters, and scene descriptions, supporting a closed-loop workflow of planning, generation, review, and local revision. Users can specify modifications through conversation, and the Agent only regenerates the problematic local shots without rerunning the entire video, transforming static generation into dynamic collaboration.
• Lightweight real-time audio-video super-resolution: To meet the high-definition requirements of professional content production, JoyAI-Echo is equipped with a single-step audio-video super-resolution module that can sharpen the basic 736×1280 output in real time to 1152×1920 or 1472×2560 under streaming latency constraints, ensuring that high-resolution output does not break the real-time nature of streaming generation.

How to Use JoyAI-Echo

• Clone the repository：git clone https://github.com/jd-opensource/JoyAI-Echo.git
• Create the environment: Use Python 3.11 + PyTorch 2.8 + CUDA 12.8, install dependencies via conda or uv, and ensure that ffmpeg is available.
• Download the model weights: Download the approximately 46GB echo-longvideo-release.safetensors and the approximately 24GB gemma-3-12b text encoder from Hugging Face, and place them in the checkpoints/ directory.
• Write the story prompts: Create a JSON file, describing each shot in the order of character and subject, action and dialogue, style, camera movement, background, sound effects and BGM.
• Run inference: Execute python inference.py; after the model is loaded in one go, it processes all prompt files and outputs to the inference_result/outputs/ directory.

Core advantages of JoyAI-Echo

• Ultra-long consistency: In videos up to 5 minutes long, character identity, visual appearance, and voice timbre remain highly consistent, completely solving the problem of the same person turning into another person as the video goes on.
• Ultra-fast generation: Memory-driven post-training combined with DMD technology increases inference speed by about 7.5 times, changing it from waiting half a day to producing a video in seconds.
• Conversational interactive creation: The Director Agent transforms static generation into dynamic collaboration, supporting natural language planning, review, and local revisions, greatly lowering the barrier to creation.
• High-definition real-time output: The lightweight super-resolution module stably outputs high-resolution video under streaming latency, meeting the needs of professional content production.
• Fully open source: The code and weights are all open source, built on LTX-2.3 and Gemma, and support academic research and secondary development.

JoyAI-Echo project address

• Project website：https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/
• GitHub repository：https://github.com/jd-opensource/JoyAI-Echo

Comparison of JoyAI-Echo with similar competitors

Application scenarios of JoyAI-Echo

• Virtual story creation and anime production: Generate coherent animated stories lasting several minutes, maintaining a high degree of consistency in character appearance, voice, and personality across multiple shots, greatly reducing traditional animation production costs.
• Digital human content production and livestreaming: Quickly generate long video content for virtual streamers and digital human customer service, ensuring that the digital human's face and voice do not drift during long-duration output, enhancing realism and professionalism.
• Rapid iteration of brand marketing videos: Through the Director Agent's conversational editing capability, marketing teams can modify advertising scripts and shots as easily as chatting, quickly producing multiple versions of brand videos and shortening the creative cycle.
• Film and television previsualization and storyboard production: Directors and producers can use natural language to generate feature-length storyboards and previsualization videos, validating shot language, character blocking, and narrative rhythm before formal shooting, reducing trial-and-error costs.