Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation

Steven Xiao, Xindi Zhang, Dechao Meng, Qi Wang, Peng Zhang, Bang Zhang · 2025

Read Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation by Steven Xiao, Xindi Zhang, Dechao Meng, Qi Wang, Peng Zhang, Bang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21734.txt.

Introduction

Steven Xiao, Bang Zhang, and their co-authors present Knot Forcing, a streaming framework that animates a portrait in real time, indefinitely. Interactive avatars and virtual assistants need animation that is sharp, smooth, fast, and responsive to changing inputs. Diffusion models give quality but are not built for streaming. Autoregressive generators can stream frame by frame, yet errors accumulate, motion jumps at chunk boundaries, and long-term consistency fades.

Knot Forcing generates video in chunks while protecting identity and continuity. It caches key-value states of the reference image to hold identity, and uses sliding-window attention locally. A temporal knot module overlaps neighboring chunks and passes space-time cues between them, smoothing the seams. A running-ahead mechanism keeps the frame's time coordinate ahead of the current frame, anchoring coherence. The result is high-fidelity animation over unlimited length, in real time on consumer-grade graphics cards.

Catalog record

CREATORSteven Xiao, Xindi Zhang, Dechao Meng, Qi Wang, Peng Zhang, Bang Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-21734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation Steven Xiao, Xindi Zhang, Dechao Meng, Qi Wang, Peng Zhang, Bang Zhang Abstract Real-time portrait animation is essential for interactive applications such as virtual assistants and live avatars, requiring high visual fidelity, temporal coherence, ultra-low latency, and responsive control from dynamic inputs like reference images and driving signals. While diffusion-based models achieve strong quality, their non-causal nature hinders streaming deployment. Causal autoregressive video generation approaches enable efficient frame-by-frame generation but suffer from error accumulation, motion discontinuities at chunk boundaries, and degraded long-term consistency. In this work, we present a novel streaming framework named Knot Forcing for real-time portrait animation that addresses these challenges through three key designs: (1) a chunk-wise generation strategy with global identity preservation via cached KV states of the reference image and local temporal modeling using sliding window attention; (2) a temporal knot module that overlaps adjacent chunks and propagates spatio-temporal cues via image-to-video conditioning to smooth inter-chunk motion transitions; and (3) A “running ahead" mechanism that dynamically updates the reference frame’s temporal coordinate during inference, keeping its semantic context ahead of the current rollout frame to support long-term coherence. Knot Forcing enables high-fidelity, temporally consistent, and interactive portrait animation over infinite sequences, achieving real-time performance with strong visual stability on consumer-grade GPUs. Project page: https://humanaigc.github.io/knot_forcing_demo_page/ . 1 Introduction Figure 1 : Streaming video generation setup. (a) T2V causal video diffusion. (b) Our approach for portrait animation: given a reference frame, we generate video autoregressively with a short sliding attention window, ensuring low latency, balanced computation, and stable identity preservation. Real-time portrait animation 10 ; 33 ; 24 ; 40 has become a cornerstone of interactive digital experiences, powering applications such as virtual assistants, live avatars, and immersive communication systems. Recent advances in diffusion transformer (DiT)-based video generation 31 ; 21 ; 30 ; 17 ; 9 ; 11 have provided a powerful engine for high-fidelity, photorealistic, and expressive digital human synthesis. Equipped with strong generative priors and sophisticated temporal modeling, these models support flexible conditioning on diverse user inputs—such as reference images, human poses, facial expressions, audio, or text—enabling highly controllable and personalized portrait animation 8 ; 6 ; 18 ; 15 ; 24 . However, these models typically operate on long token sequences and rely on iterative denoising, leading to high computational cost and slow generation speed. As a result, they struggle to meet the real-time demands of interactive applications, where low latency and immediate response are critical. This limits their practicality for on-the-fly portrait animation, despite their high visual quality. Recently, causal autoregressive (AR) video generation model has emerged as a promising alternative for low-latency inference. By leveraging score distillation 37 ; 39 , these works 39 ; 13 ; 7 ; 36 ; 19 ; 22 effectively transfer the rich contextual understanding of bidirectional teacher models into causal AR generators. Furthermore, by employing key-value (KV) caching and reducing the number of denoising steps, these models generate video frames in a streaming fashion (illustrated in Fig. 1 (a)), significantly reducing real-time latency while preserving temporal coherence. However, these methods primarily focus on open-ended tasks such as text-to-video generation, and still suffer from noticeable visual degradation, flickering artifacts, and temporal inconsistency. In this work, we present Knot Forcing , a novel streaming framework tailored for real-time portrait animation, designed to balance quality, efficiency, and interactivity. Our approach integrates three key innovations: (1) We propose a chunk-wise causal generation framework that autoregressively produces video in manageable segments, balancing computational latency and temporal coherence, while supporting seamless integration of streaming control signals (e.g., poses or audio) for real-time controllability. To maintain identity consistency, we encode the user-provided reference image using the video DiT and cache its KV states as a global semantic anchor, which is fused with streaming video features throughout generation. A short sliding window attention is further applied to model local temporal dependencies between adjacent chunks (shown in Fig. 1 (b)). (2) We observe that, unlike bidirectional models which provide each token with a consistent, full-sequence attention context, causal autoregressive video models suffer from periodic shifts in attention context between adjacent frames. This leads to visual drift and motion discontinuities at chunk boundaries, resulting in noticeable degradation in temporal stability and visual fidelity compared to the bidirectional teacher. To address this, we propose the Temporal Knot module, which introduces a temporal overlap between the tail frames of the previous chunk and the head frames of the current chunk. By leveraging image-to-video (I2V) conditioning, it propagates fine-grained spatio-temporal cues across chunk boundaries, effectively bridging semantic gaps and ensuring smooth motion transitions. This design enhances both local and global temporal consistency, enabling the causal model to better approximate bidirectional dynamics while preserving streaming efficiency and delivering high-fidelity portrait animation. (3) To address error accumulation in infinite portrait animation, we propose global context running ahead, which maintains a forward-looking goal state during synthesis. During training, the model learns from short clips with the final frame serving as a future-facing reference. At inference, we treat the ground-truth reference frame as a moving “pseudo-final" frame, dynamically updating its temporal position according to the current generation step—by adjusting its rotary positional encoding (RoPE) and re-caching its KV states. This ensures the reference remains temporally ahead, providing a stable structural and identity prior that continuously guides streaming predictions toward the target trajectory. The approach effectively suppresses error propagation, preserves motion diversity, bridges the short-to-long gap between training and inference, and enhances visual fidelity in long-term generation. We validate our method on diverse real-time, controllable portrait animation tasks under long-horizon setting. Experiments show that Knot Forcing outperforms existing approaches in visual stability, temporal coherence, and generation quality, with significantly reduced flickering, motion jitters, and identity drift. The framework maintains low latency and strong responsiveness to streaming controls, enabling high-fidelity, infinite-duration animation. Ablation studies confirm the contribution of each component, and overall, our method advances the state of the art in real-time, controllable portrait animation. 2 Preliminaries 2.1 Autoregressive Video Diffusion Current video diffusion transformers 21 ; 30 ; 17 ; 9 ; 11 represent videos as extended token sequences, applying denoising across the full sequence. Although this yields videos with high coherence and quality, the high inference latency makes it difficult to adapt for real-time generation demands. Recently, the development of autoregressive (AR) video diffusion models 3 ; 27 ; 36 ; 19 ; 7 ; 13 ; 39 offers a promising avenue for streaming generation. These models represent a hybrid approach, effectively combining the autoregressive chain-rule decomposition with the power of denoising diffusion models for video synthesis. Given a sequence of N video frames x^{1:N}=(x^{1},x^{2},...,x^{N}) , AR models factorize the joint distribution into product of conditionals using the chain rule: \displaystyle p(x^{1:N})={\textstyle\prod_{i=1}^{N}}p(x^{i}|x^{<i}), (1) where each conditional distribution p(x^{i}|x^{<i}) is modeled via a diffusion process, wherein each frame x^{i} is synthesized by progressively denoising Gaussian noise, conditioned on the preceding frames x^{<i} . Practically, many models 27 ; 39 choose to model conditional distributions chunk by chunk, predicting c future frames concurrently based on the current generated frames. During inference, AR video diffusion models leverage KV caching to store the historical context x^{<i} (shown in Fig. 1 (a)). This allows

Read the complete text