Wan-Streamer v0.2 - Alibaba Tongyi's Omni-modal Understanding and Generation Model

What is Wan-Streamer v0.2

Wan-Streamer v0.2 is an end-to-end omni-modal understanding and generation model for real-time duplex interaction launched by Alibaba Tongyi Lab. The model unifies "listening, seeing, speaking, and performing" into a single Transformer, natively supporting real-time understanding and synchronous generation of text, audio, and video, with an end-to-end response latency of only 550ms and an output resolution of 640×368@25FPS, enabling AI to listen, watch, and respond like a real person, achieving truly natural face-to-face video communication.

Main Features of Wan-Streamer v0.2

• Real-time audio-video conversation: Supports video-call-style face-to-face interaction, with AI perceiving the user's audio and video in real time and synchronously generating responses.
• Omni-modal understanding: Natively supports real-time understanding of text, audio, and video inputs without the need for external module assembly.
• Synchronous audio-video generation: Synchronously outputs speech and high-definition video frames, achieving perfect synchronization between sound and picture.
• Micro-expression and gesture generation: Can generate natural eye contact, body posture, gestures, and scene details.
• Free role-playing: Supports real-time generation of any character for conversation through natural language description.

Technical Principles of Wan-Streamer v0.2

• Native streaming architecture: Unifies user input and agent output onto the same causal timeline, without the need to wait for the entire utterance to finish.
• Streaming unit closed loop: Completes a full loop of perception, understanding, generation, and decoding every 160ms, enabling speaking, listening, and responding simultaneously.
• Thinker-Performer dual pathway: A single-card Thinker is responsible for low-latency perception and language reasoning, while a multi-card Performer with Ulysses parallelism is responsible for high-definition video generation.
• Temporal overlapping scheduling: The computation windows of the single-card Thinker and multi-card Performer overlap, removing video generation cost from the latency-sensitive path.
• End-to-end unified modeling: Unifies text, audio, and video inputs and outputs into a single Transformer, avoiding pipeline fragmentation.

How to Use Wan-Streamer v0.2

Currently, Wan-Streamer v0.2 is released as a research project. For specific access portals and API integration methods, please follow the subsequent official announcements from Tongyi Lab.

Core Advantages of Wan-Streamer v0.2

• Extreme low latency: 550ms end-to-end, with only 200ms on the model side, significantly faster than mainstream real-time voice conversation models.
• Omni-modal end-to-end: Natively supports synchronous audio-video understanding and generation, without the need for external modules such as ASR, LLM, and TTS.
• High-definition output: 640×368@25FPS, supports micro-expressions, gestures, and scene details, no longer limited to a floating head.
• Full-duplex interaction: Speaks, listens, and responds simultaneously, without waiting for the user to finish speaking before processing, making communication more natural.
• Scalable architecture: The Thinker-Performer dual-pathway design improves image quality while maintaining extremely low latency.

Project Address of Wan-Streamer v0.2

• Project official website：https://wan-streamer.com/
• arXiv technical paper：https://arxiv.org/pdf/2607.04443

Comparison of Wan-Streamer v0.2 with Similar Competing Products

Application scenarios of Wan-Streamer v0.2

• Video call AI assistant: Open the camera for face-to-face communication, suitable for scenarios requiring a “sense of presence” such as spoken language practice, interview simulation, and psychological counseling.
• Scenario-based education: The AI teacher can observe students’ facial expressions through the screen to judge their level of understanding and adjust the teaching pace in real time.
• Immersive game NPC: Game characters have facial expressions, body language, and real-time reactions, engaging in truly “face-to-face” conversations with players.
• Accessible interaction: Generate real-time video responses with precise lip movements and gestures for hearing-impaired users, and describe the surrounding environment for visually impaired users.
• Virtual role-playing: Enables real-time conversations with historical figures, artistic characters, or custom roles, delivering an immersive cultural experience.