Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

Wan Team, Alibaba Group · 2026

Read Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models by Wan Team, Alibaba Group (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25041.txt.

Introduction

The Wan Team at Alibaba Group presents Wan-Streamer, a foundation model built from the ground up for real time, full duplex audio visual interaction. Usual interactive systems chain separate modules: voice detection, speech to text, a language model, speech synthesis, and video generation. Each handoff adds latency and error. Wan-Streamer instead uses a single transformer for everything. Language, audio, and video are modelled as both input and output, with interleaved tokens and block causal attention for stepwise streaming. Perception, reasoning, generation, response timing, and turn management are learned jointly.

The whole stack is redesigned for streamability, with causal encoders and decoders and low latency token scheduling. Streaming units run as short as 160 milliseconds at 25 frames per second. Model side response latency is about 200 milliseconds. Total latency stays near 550 milliseconds even with a realistic network delay included, supporting sub second two way audio visual communication.

Catalog record

CREATORWan Team, Alibaba Group
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25041

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models Wan Team, Alibaba Group Abstract We present Wan-Streamer , a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer , where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction. ] See Contributions and Acknowledgements for the full author list. \metadata [Website] https://wan-streamer.com/ 1 Introduction Human interaction with the physical world is fundamentally streaming and full-duplex. People do not first finish perceiving, then reason in isolation, and only afterwards produce a response. Instead, they continuously watch, listen, speak, gesture, react, pause, and interrupt, with perception and expression overlapping at audio-visual timescales. Building artificial systems with the same interaction pattern is becoming increasingly important for embodied assistants, real-time digital humans, live broadcasting, interactive entertainment, and world models that can be explored or controlled online [ 22 , 39 , 14 ] . These applications require more than a model that can understand an image, generate a clip, or answer a text prompt. They require a real-time interactive foundation model: a model that continuously consumes audio-visual observations, maintains a persistent world and dialogue state, decides when and how to respond, and expresses that response through synchronized language, speech, and video with very low latency. Recent progress has advanced several pieces of this goal. Multimodal language models can reason over visual and acoustic inputs [ 3 , 35 , 15 ] , while large-scale video generation models can synthesize increasingly realistic motion and appearance from text, audio, camera trajectories, or actions [ 37 , 31 , 44 , 14 , 2 , 39 ] . Causal and streaming generation methods further make incremental synthesis feasible by replacing offline bidirectional generation with cached context and rolling prediction [ 17 , 7 , 25 ] . However, these advances are still usually assembled as asymmetric or cascaded systems. Some systems perceive audio and video but respond only in text or speech; others generate audio-visual behavior but rely on external language, ASR, TTS, animation, or rendering modules. Even when the interface appears multimodal, text is often used as a hidden intermediate representation between separately trained components [ 28 , 6 ] . Such pipelines introduce waiting time at module boundaries, accumulate recognition and synchronization errors, and make response timing, turn management, identity preservation, and long-horizon consistency difficult to learn as part of one behavior. The core difficulty is that real-time audio-visual interaction is not simply the union of multimodal understanding and multimodal generation. It is intrinsically full-duplex rather than a simple alternation between input and output: when the user is speaking, the agent should still produce visible listening behavior, and when the agent is responding, it should still perceive the user’s audio-visual feedback for interruption and adaptation. Different modalities have different token rates, representations, objectives, and latency constraints, yet they must be causally aligned within a single ongoing process. Incoming speech and video should immediately affect outgoing speech and motion; generated audio and visual states should be coupled before decoding, not repaired afterwards; and every emitted unit should become part of the full interaction history so that the model can preserve identity, scene state, speaking rhythm, and user intent over long sessions. These requirements make streamability a modeling constraint rather than a serving optimization. A system designed around offline encoders, bidirectional video decoders, round-based dialogue, or post-hoc audio-visual synchronization cannot recover truly low-latency full-duplex behavior by engineering alone. Figure 1 : Overview of Wan-Streamer. It models language, audio, and video as both input and output within a single Transformer , using block-causal attention for incremental streaming generation. To address this challenge, we design Wan-Streamer from the ground up as a native-streaming, end-to-end model for real-time full-duplex audio-visual interaction, as illustrated in Fig. 1 . Wan-Streamer is built around one streaming contract: every component must operate causally, every newly observed unit must be usable immediately, and every generated unit must be emitted and committed back into the interaction history. Wan-Streamer represents language, audio, and video, on both the input and output sides, as an interleaved causal sequence processed by a single Transformer . The model does not rely on external VAD, ASR, language, TTS, audio-driven animation, or video-generation modules. Instead, audio-visual perception, semantic reasoning, response planning, speech generation, visual generation, response timing, and turn-taking behavior are optimized jointly within one persistent interaction state. To make this possible, the entire stack is designed for causality from the beginning: strictly causal audio and video variational autoencoders (VAEs) for streaming latent coding, causal audio-visual encoders, causal audio and video decoders, and a temporally causal Transformer coordinated by block-causal attention. During inference, Wan-Streamer remains a single end-to-end model, but we deploy it as a thinker-performer streaming pipeline to maximize overlap and hardware utilization. At streaming step k , the thinker first consumes the current user audio-visual observations, applies the causal audio-visual encoders, and runs the short token-causal Transformer pass for language prediction and state update. This produces the new KV-cache slice for the current interaction state. At the communication boundary, the thinker receives the audio and video latents generated by the performer for the previous response unit, and sends the current KV-cache slice to the performer. The thinker then decodes those previous latents into output audio and video for immediate emission, while the performer uses the newly received full-history KV context to run only the flow-matching solver for the next audio-visual latent unit. The resulting latents are kept on the performer and returned to the thinker at the next streaming step. This schedule preserves the unified causal state through KV exchange, but places expensive latent generation on the performer and allows current-frame perception/state update, previous-frame output decoding, next-frame latent denoising, and KV/latent communication to overlap across adjacent streaming units. Together with CUDA graph capture, compilation, and optimized kernels, Wan-Streamer reaches approximately 200 ms model-side response latency. With 350 ms bidirectional network latency, the total interaction latency is approximately 550 ms, supporting sub-second audio-visual communication without the module-boundary waiting time of typical cascaded real-time dialogue systems. Our contributions are summarized as follows: • We introduce Wan-Streamer, a native-streaming, end-to-end interactive foundation model that supports language, audio, and video as both inputs and outputs within a single Transformer, without relying on external language, speech,

Read the complete text