Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs
Sicheng Xu Yu Deng Shoukang Hu Yichuan Wang Yizhong Zhang, Zhan Chen Jiaolong Yang Baining Guo · 2026
Read Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs by Sicheng Xu Yu Deng Shoukang Hu Yichuan Wang Yizhong Zhang, Zhan Chen Jiaolong Yang Baining Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01620.txt.
Introduction
Xu and co-authors generate streamable talking portrait videos in real time. Video diffusion models make vivid portraits but cost too much compute. Interactive uses need streaming, not offline rendering. Their framework conditions on speech audio and reference images. A causal video VAE performs deep latent compression. Reference images guide the VAE as extra inputs. The network can then focus on dynamic information. Static appearance needs less capacity under that design. A residual auto-encoding extension sharpens space-time causality handling.
The generator is a Rectified Flow Transformer. It produces video latents block by block, autoregressively. Speeds beat baseline models by a significant margin. Quality matches or beats large models on realism and vividness. Comprehensive experiments back those claims.
Catalog record
| CREATOR | Sicheng Xu Yu Deng Shoukang Hu Yichuan Wang Yizhong Zhang, Zhan Chen Jiaolong Yang Baining Guo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01620
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs Sicheng Xu Yu Deng Shoukang Hu Yichuan Wang Yizhong Zhang, Zhan Chen Jiaolong Yang Baining Guo Abstract Video diffusion models have significantly advanced portrait video generation, yet their high computational demands limit their use in interactive applications. This work presents a framework for streamable talking portrait video generation conditioned on speech audio and reference images. Designed meticulously for streaming scenarios, it features a causal video VAE for deep latent compression and an autoregressive latent denoising model. Our causal VAE integrates a variable number of reference images as guidance, allowing the network to focus on dynamic information rather than static appearance, thereby enhancing compression efficacy and reconstruction quality. Additionally, we extend the residual auto-encoding paradigm to improve spatial-temporal causality handling in our VAE. The generator is based on a Rectified Flow Transformer architecture and produces video latents in a blockwise auto-regressive manner. Our method enables the real-time generation of high-quality talking portrait videos, achieving speeds significantly faster than baseline models. Furthermore, comprehensive experiments demonstrate that it is on par with or even outperforms these large models in realism, vividness, and video quality. 1 Introduction With the advent of Large Language Models such as ChatGPT 202 2024 , there has been a growing trend toward leveraging human–AI interaction tools to enhance work efficiency and interpersonal communication. Among these tools, lifelike talking portraits featuring dynamic visual representations play a crucial role in conveying subtle cues that go beyond spoken or written language. For example, they are essential for applications such as social companionship, interactive education, and mental health support, where dynamic and empathetic communication is paramount. Generating talking portrait videos from audio has been actively studied in the past. Most earlier works have focused on the face or head region Chen et al. 2018 ; Prajwal et al. 2020 ; Wang et al. 2021a ; Wang et al. 2021b ; Zhang et al. 2023 ; Wang et al. 2023 ; Xu et al. 2024b and highly-realistic talking head results have been achieved Xu et al. 2024b ; Guo et al. 2024 . However, these methods often rely on specific human face or head priors and extending them to encompass larger portrait regions like the torso is difficult. Additionally, their ability to generate non-rigid dynamics beyond facial areas and complex lighting and shadow effects is limited, preventing them from reaching a higher degree of realism. Recent advances in large diffusion models for video generation have dramatically pushed the boundaries for talking portrait video Rombach et al. 2022 ; Blattmann et al. 2023 ; Stypułkowski et al. 2024 ; Tian et al. 2024 ; Xu et al. 2024a ; Wei et al. 2024 ; Wang et al. 2025b ; Lin et al. 2025a . These large models are able to generate vivid and realistic dynamics across much larger regions beyond head and produce high-quality videos with strong aesthetic appeal. However, their high computational cost restricts their application to offline content generation, hindering their use in broader real-time and interactive streaming scenarios. For instance, state-of-the-art models require several minutes to generate a 5-second video on modern GPUs Wang et al. 2025b ; Ji et al. 2025 ; Cui et al. 2025 . Although various approaches have been proposed to improve the efficiency of video diffusion Yin et al. 2024c ; Yin et al. 2024a , adapting the large models for real-time streaming scenarios is still a formidable task. Our goal is to achieve efficient talking portrait video generation from audio. Specifically, we have three objectives: 1) real-time, streamable generation to support interactive applications; 2) high visual quality and vividness to ensure realism, including precise audio-lip synchronization, vivid facial and body motions, realistic lighting and dynamic effects, and high video quality; 3) handling a wide torso region beyond the head to improve immersiveness. Collectively, these requirements present significant challenges, particularly in achieving both efficiency and quality to a high standard, which remains largely unattainable with the existing methods. Towards these objectives, we present a new framework for streamable talking portrait video generation. Our models include a causal video VAE for latent compression and an autoregressive latent denoising model conditioned on audio. The causality of the VAE ensures smooth video generation given arbitrary-length video latents, preventing temporal discontinuities. Meanwhile, the autoregressive scheme of the denoising generator which can leverage key–value (KV) caching for efficient inference is naturally suited for streaming. To obtain a compact latent space for efficient generation, we introduce two key designs for our causal VAE. First, in contrast to ordinary VAEs, we introduce reference image guidance to the VAE decoder. The intuition is that, unlike generic video generation, talking portrait videos typically feature a fixed subject as the dominant content. The user’s reference image shares significant information with the generated videos. By incorporating it into the VAE, compression efficacy is enhanced as the network can focus less on the subject’s appearance and background, and more on extracting dynamic information. We train the VAE to accept one or multiple reference images and they significantly improve reconstruction quality. Second, we extend the residual auto-encoding paradigm of DC-AE Chen et al. 2024b to our causal video VAE. Temporal and spatial residual encoding are applied in separate steps when resolution changes, with the first frame handled individually to preserve causality in temporal processing. This design also leads to significantly enhanced reconstruction fidelity. Given the learned compact latent vectors, we apply a Rectified Flow Transformer Lipman et al. 2022 ; Esser et al. 2024 as the generator conditioned on the audio input and the reference image(s). The generator is trained in a blockwise autoregressive manner with a teacher-forcing strategy Zhou et al. 2025 . We train our VAE and generator on a corpus of talking portrait videos. Comprehensive experiments show that our method is not only significantly faster than the baseline portrait video generation models but also producing comparable or even better results than these models across various metrics of realism and vividness. Our main contributions are summarized as follows: • We present a framework for streamable talking portrait video generation with a block-wise autoregressive denoising model and a causal video decoder, which can continuously generate arbitrary-length video from audio. • We propose a reference-guided video VAE with a causal residual video auto-encoding scheme adapted from DC-AE Chen et al. 2024b , which can incorporate one or more reference portrait images into decoder to attain significantly enhanced video reconstruction fidelity. • We implement a video compression rate of 768, approximately 10-15 \times higher than those of the VAE in popular video diffusion models Yang et al. 2024 ; Wan et al. 2025 , and we achieve a video generation speed of 42 FPS, over 25 \times faster than existing diffusion-based talking portrait generation model Wang et al. 2025b ; Ji et al. 2025 ; Cui et al. 2025 , and meanwhile comparable or higher quality results compared to these models across various metrics. 2 Related Work Audio-driven talking portrait generation. Generating talking head videos from audio input has been a long-standing problem in computer vision and graphics. Early works primarily focus on synthesizing lip movements synchronized with the input speech Brand 1999 ; Bregler et al. 2023 ;