EchoWM - JD's Open-Source Interactive Audio-Visual World Model

What is EchoWM

EchoWM is an interactive audio-visual world model open-sourced by JD Exploration Academy. The model continuesJoyAI-Echo's native audio-visual generation capabilities, allowing users to move and explore in real time in an AI-generated world from a first-person/third-person perspective, with visuals, spatial relationships, ambient sound, and speech continuously synchronized and not drifting over long periods. The model is trained with a unified camera intent interface and a world data engine, ranking first on the WBench navigation benchmark with an average score of 81.7.

Main Features of EchoWM

• Real-time explorable world generation: Users can freely move, turn, and backtrack in AI-generated scenes using the WASD directional keys, and the world unfolds in real time with the operations.
• Native joint audio-visual generation: Synchronously generates 720p video and ambient sound, music, and character speech within the same framework, with sound naturally continuing as the visuals change.
• First/third-person dual-perspective control: In first-person, the camera is the observer moving; in third-person, the camera automatically follows the movement of subjects such as characters and vehicles.
• Multi-round long-horizon continuation: Uses the audio-visual content at the end of the previous segment as the context for the next round; after multiple rounds of operations, the scene layout, subject appearance, and sound remain coherent.
• First on the WBench navigation benchmark: Average score of 81.7 across 158 cases, ranking among the top in both consistency and interactivity metrics.

Technical Principles of EchoWM

• Unified camera intent interface: Maps discrete keyboard commands into continuous 6-DoF trajectories relative to the initial pose, using the same interface to describe how the camera moves; in first-person it directly represents observer motion, while in third-person the model learns the "subject movement—camera following" coupling relationship from data, without the need for additional character trajectories or camera preset parameters, and the trajectory is injected only into the video branch, with audio linked to the visuals.
• World data engine: Integrates four complementary data sources—internally collected gameplay, human real-play recordings, UE simulation, and ordinary internet videos; then uses the 90th percentile of the maximum trajectory translation magnitude as a global "ruler" for unified normalization, preserving the relative displacement differences between different trajectories and avoiding speed discontinuities caused by segment-wise calibration.
• Progressive four-stage training: The first stage uses AV-rich data for audio-visual continual pretraining as the foundation; the second stage freezes the backbone and trains only a lightweight trajectory branch to strengthen control responsiveness; the third stage performs joint fine-tuning with a low learning rate using data that has both control reliability and audio-visual quality; the fourth stage is autoregressive post-training, transformed into causal chunked generation through Teacher Forcing, combined with short-horizon Self-Gradient Forcing to let the model adapt to its own generated history, then compressed into four-step sampling through distribution matching distillation, and paired with a sink-plus-FIFO caching mechanism to support long-horizon streaming interaction while controlling KV cost.

Follow WeChat and reply "open source" to join theAI Open-Source Project Exchange Group

How to Use EchoWM

• Clone the repository：git clone Download github.com/jd-opensource/JoyAI-Echo, enter the echo_wm/ directory (independent of the long-video project echo_longvideo/ ).
• Create the environment：conda create -n echo-wm python=3.11 Create an independent environment and activate it.
• Install dependencies: First install PyTorch 2.9.1 (cu128), then pip install -r requirements.txt, and use torch.cuda.is_available() to verify that the GPU is available.
• Download model weights: Use hf download Echo-Team/Echo-WM to download the model checkpoint; you also need to download the Gemma 3 text encoder (a gated repository; you must first accept the license on the HF page and hf auth login）。
• Run the official examples to verify: Run scripts/run_wm_case.py --case examples/wm_cases/0010 and other built-in complete examples to confirm that the environment is configured correctly.

• Custom generation: Run inference_wm.py, pass in the first-frame image + six-field prompt + action string (Action DSL) to output a video with sound.
• Write action instructions: Use Action DSL to describe camera movement, with the format <按键>-<帧数>, connected by commas, for example w-60,a-60 means moving forward for 60 frames then moving left for 60 frames, supporting combinations of w/s/a/d/i/j/k.

EchoWM's core advantages

• Integrated audio-visual: Natively jointly generates 720p video and ambient sound, music, and speech within the same framework, with sound changing in sync with the scene and events in the visuals, solving the problem that most world models "only produce visuals but no sound."
• Unified control interface: Uses a single set of relative 6-DoF trajectories to simultaneously drive first-person observer movement and third-person camera following of the subject, with no need to design separate controllers for different viewpoints.
• Long-horizon coherence: Through autoregressive post-training and the sink-plus-FIFO cache mechanism, users can continue multi-round exploration based on the audio-visual context at the end of the previous round, with scenes, subjects, and sound staying on track over long periods.
• Leading in evaluation: Ranked first on the WBench Navigation evaluation with an average score of 81.7, and in user studies overall preference was significantly better than LingBot-World-v2 and HappyOyster.

JoyAI-EchoWM project address

• Project website：https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/wm/
• GitHub repository：https://github.com/jd-opensource/JoyAI-Echo
• arXiv technical paper：https://arxiv.org/pdf/2608.23189

Comparison of EchoWM with similar competing products

EchoWM's application scenarios

• Game prototyping: Creators enter generated scenes to playtest, verifying spatial layout, route design, and camera experience before deciding on the production direction.
• Interactive storytelling: The plot advances along branches according to user choices and is continuously generated, replacing traditional pre-made assets, for use in interactive dramas and interactive fiction.
• Immersive content: Film, television, and advertising content shifts from "watching" to "entering," allowing viewers to freely explore the world within the frame.
• Virtual tours: Low-cost digital roaming for museums, scenic areas, real estate, and other scenes, without the need for frame-by-frame modeling.
• Digital human performances: Character speech, movements, and the surrounding dynamic environment, camera, and sound are generated in coordination, suitable for virtual streamers and digital employees.