AdaState: Self-Evolving Anchors for Streaming Video Generation
Yusuf Dalva, Pinar Yanardag · 2026
Read AdaState: Self-Evolving Anchors for Streaming Video Generation by Yusuf Dalva, Pinar Yanardag (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30349.txt.
Introduction
Yusuf Dalva and Pinar Yanardag replace the static anchor in autoregressive video diffusion with an adaptive state, a hidden latent the model denoises with content per chunk but never renders. This anchor draws excess attention, damping video motion and locking scene layout to the initial viewpoint as scenes evolve. Rather than using a frozen first frame, the model makes its own scene anchor per step by looking at the previous state and current content. Tests show the adaptive state greatly boosts video motion, giving richer motion and natural scene flow.
Catalog record
| CREATOR | Yusuf Dalva, Pinar Yanardag |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57231 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30349
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdaState: Self-Evolving Anchors for Streaming Video Generation Yusuf Dalva, Pinar Yanardag Abstract Autoregressive video diffusion models generate streaming video by producing frames sequentially, conditioning each chunk on previously generated content. These models are structurally anchored to the first frame: its key-value representation occupies a privileged position in the attention cache and serves as the primary scene reference throughout generation. As the cleanest and most error-free position in the cache, this anchor draws disproportionate attention, suppressing video dynamics, and locking scene composition to the initial viewpoint even as the scene naturally evolves. The result is a temporally shallow video in which motion, camera movement, and scene progression are dampened in favor of static consistency. To address this, we replace the static anchor with an adaptive state , a hidden latent that the model denoises alongside content at every chunk but never renders. Rather than referencing a frozen first frame, the model generates its own scene anchor at each step by attending to both the previous state and the current content, producing a reference that evolves with the generated content. Unlike standard video generation, which encodes an absolute notion of time, our formulation treats time as relative: every generation step sees the same positional structure regardless of how far generation has progressed, and the state transition is identical at every chunk. Together, these properties introduce a recurrence into the generation process, where denoising serves as the transition function, and the KV cache serves as the carrier, requiring no external module. Experiments demonstrate that the adaptive state substantially improves video dynamics, enabling richer motion and natural scene progression within generated videos. Figure 1: AdaState. Colored markers highlight the scene at each timestamp; dashed lines trace their progression ( red : baselines, teal : AdaState). Top, t=30s (6 \times training horizon): Infinity-RoPE’s static anchor cannot adapt to the evolving scene, forcing the model to realize all implied content, schools of fish, sea turtles, within the initial layout, producing hallucinated duplications by t=30. AdaState’s markers drift forward as new terrain and subjects emerge naturally across the rollout. Bottom, t=5s: Self-Forcing holds the scene in place while neon lighting dims and the cityscape desaturates. AdaState produces continuous scene evolution while preserving visual characteristics. 1 Introduction Autoregressive video diffusion models generate streaming video by producing one chunk of frames at a time, conditioning each chunk on previously generated content [ 12 , 32 , 28 , 16 , 30 ] . These models are structurally anchored to the first frame: its key-value representation occupies a privileged position in the attention cache and serves as the primary scene reference throughout generation, exploiting the attention-sink phenomenon [ 26 ] where causal softmax concentrates mass on initial positions. However, as the cleanest and most error-free position in the cache, this static anchor draws disproportionate attention, suppressing video dynamics and locking scene composition to the initial viewpoint even as the scene naturally evolves during generation. Motion, camera movement, and scene progression are dampened in favor of static consistency, producing temporally shallow video that lacks the dynamic richness of natural video (see Figure 1 ). The static anchor also limits the model’s trained robustness, since errors accumulated over the autoregressive rollout are absorbed by the clean reference rather than surfaced at the model’s most attended position, leaving the training objective with limited leverage to shape behavior under the imperfect conditions the model will encounter as generation extends. Existing approaches do not address this root cause: static sinks [ 28 , 16 ] retain first-frame tokens as fixed anchors, reinforcing the shortcut. EMA-based methods [ 17 , 14 ] apply content-agnostic averaging over evicted content, converging to a blurry mean that cannot adapt to scene changes. Token replacement approaches [ 15 ] substitute raw cached content into the anchor position on a heuristic schedule, keeping the reference fresh but biasing generation toward reproducing past frames rather than producing novel continuation. All of these either preserve the static anchor or update it through operations external to the generative model, leaving the fundamental attention bias intact. In this paper, we replace the static anchor with an adaptive state , a hidden latent that the model denoises alongside content at every chunk but never renders. At each step, the model generates its own scene anchor by attending to both the current content and the previous adaptive state, producing a reference that evolves with the generated content rather than remaining locked to the initial frame. After denoising, the state’s clean representation is written to the anchor position in the cache, while the content is decoded into video; the state is carried forward silently as an evolving scene reference. Unlike standard video generation, which encodes absolute temporal position, our formulation treats time as relative: every generation step sees the same positional structure regardless of how far generation has progressed, removing the notion of a privileged time zero. This design reveals that denoising is itself a recurrence: the adaptive state is a hidden variable updated by the model’s own iterative refinement and carried via the KV cache, turning sequential autoregressive generation into a recurrent process with no external module or gating mechanism. To ensure the training objective emphasizes the frames that depend most on the adaptive state, we further propose horizon-weighted DMD, a per-frame loss weighting that increases with frame index, preventing the optimizer from concentrating capacity on clean, early frames. Experiments demonstrate that the adaptive state substantially improves video dynamics, enabling richer motion and natural scene progression within generated videos. Our contributions: (1) identifying the structural attention bias that suppresses dynamics in autoregressive video diffusion; (2) replacing the static anchor with an adaptive state trained via horizon-weighted DMD; and (3) showing that denoising is itself a recurrence, a hidden latent carried via the KV cache with no external module. Figure 2: The anchor-recency structure of streaming video attention. (a) Off-diagonal attention in Self-Forcing across chunk depths. The anchor at position 0 (squares) and the freshest chunk frame (triangles) consistently dominate; remaining positions receive roughly uniform mass. (b) 5-second generation on the same prompt. Without a persistent reference, coherence degrades over time. A static reference preserves identity but freezes the scene. AdaState maintains identity while the camera moves and the environment evolves. 2 Related Work Streaming autoregressive video diffusion. Autoregressive video diffusion models generate chunks sequentially with causal attention and KV caching for bounded-cost streaming. Distillation-based training has been the central thread: CausVid [ 32 ] introduced asymmetric distillation from bidirectional teachers, Self-Forcing [ 12 ] trains on model outputs via DMD to close the train-inference gap, Self-Forcing++ [ 4 ] extends this to minute-scale, and Causal Forcing [ 35 ] refines the per-frame objective. Rolling Forcing [ 16 ] jointly denoises a rolling window, Reward Forcing [ 17 ] adds reward-weighted distillation, and MMM [ 2 ] couples flow matching on long videos with distribution matching on sliding windows. A common design retains first-frame KV as a static attention anchor, exploiting the attention-sink phenomenon [ 26 ] ; LongLive [ 28 ] makes this explicit by pinning the first frame’s KV as a permanent sink. Approaches that update the anchor are content-agnostic: EMA over evicted tokens (Reward Forcing), dual-rate EMA with online RoPE re-indexing (MemRoPE [ 14 ]