Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation
Jiayi Luo Qiyan Liu Tengyang Wang Junhao Liu Jiayu Chen Cong Wang, Hanxin Zhu Chen Gao Xiaobin Hu Qingyun Sun Zhibo Chen, [0.6em] BUAA · 2026
Read Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation by Jiayi Luo Qiyan Liu Tengyang Wang Junhao Liu Jiayu Chen Cong Wang, Hanxin Zhu Chen Gao Xiaobin Hu Qingyun Sun Zhibo Chen, [0.6em] BUAA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30083.txt.
Introduction
Jiayi Luo and ten colleagues propose Future Forcing, a training-free cache policy for autoregressive video generation that plans ahead. Standard cache pruning judges tokens by current or past context, discarding tokens that look minor early but matter later. Pre-RoPE query patterns stay stable across steps, so future queries can be estimated from past statistics. Their policy builds a query proxy, scores cached tokens, and merges pairs. It improves long-horizon and subject consistency on VBench-Long for minute-long clips.
Catalog record
| CREATOR | Jiayi Luo Qiyan Liu Tengyang Wang Junhao Liu Jiayu Chen Cong Wang, Hanxin Zhu Chen Gao Xiaobin Hu Qingyun Sun Zhibo Chen, [0.6em] BUAA |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64198 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30083
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation Jiayi Luo Qiyan Liu Tengyang Wang Junhao Liu Jiayu Chen Cong Wang, Hanxin Zhu Chen Gao Xiaobin Hu Qingyun Sun Zhibo Chen, [0.6em] BUAA Abstract Autoregressive (AR) video generation has emerged as a promising paradigm for long-horizon video synthesis, where each frame is generated conditioned on previously generated tokens. To accelerate inference, the KV cache is used to avoid redundant recomputation across generation steps. Nevertheless, its growth with generation length introduces increasing memory and error accumulation, limiting the scalability of AR models to even longer sequences. Existing KV cache compression methods mitigate this issue by selectively retaining only video tokens deemed important. However, most existing methods assess token importance using short-horizon signals derived from the current or historical generation context, making these methods prone to overlooking tokens that appear unimportant at early steps but later become critical for future frames. In this work, we identify an important property of trained AR video models: although RoPE-modulated queries evolve across autoregressive steps, the underlying canonical pre-RoPE query distribution remains remarkably stable throughout the video generation process . This approximate stationarity implies that future query distributions are estimable from historical statistics, enabling principled future-aware cache decisions without any additional training. Building on this insight, we propose Future Forcing , a training-free future-aware KV cache policy for AR video generation. Specifically, Future Forcing first constructs a future query proxy from historical statistics, then scores KV cache tokens by their importance under this proxy, and finally merges redundant token pairs within the affine subspace induced by the future query. Extensive experiments show that Future Forcing improves long-horizon consistency under limited KV caches, achieving up to 1.49 improvement in subject consistency on VBench-Long for 60s generation over existing AR video KV cache policies. 1 Introduction Bidirectional Diffusion Transformer (DiT)-based video generation models Wan et al. 2025 ; Kong et al. 2024 ; Wu et al. 2025a ; Chen et al. 2025 ; Hong et al. 2023 employ bidirectional attention to jointly denoise all frames, enabling coherent full-sequence modeling and high-quality synthesis. Yet this paradigm is inherently offline, ill-suited for streaming or continual video extension Huang et al. 2025 ; Yin et al. 2025 . By contrast, autoregressive (AR) video models Zhu et al. 2026 ; Liu et al. 2025a causally condition each frame on prior tokens, enabling efficient incremental generation. To make incremental decoding efficient, AR video models reuse historical key and value representations through KV caching, avoiding repeated computation over the full history at each step. Since the cache grows with generation length, practical memory constraints require operating under a bounded KV cache budget. Under this constraint, cache selection becomes critical: evicting tokens needed by future frames distorts attention over historical context, and these errors propagate through subsequent autoregressive steps, accumulating into visual degradation, temporal inconsistency, and motion drift. However, most existing KV cache compression methods for AR video generation estimate token importance from current or recent context-derived signals Yi et al. 2025 ; Zhu et al. 2026 ; Liu et al. 2025a ; Lu et al. 2025 ; Huang et al. 2025 . These signals reflect present relevance but cannot anticipate future attention demands, as future query representations are unavailable at KV cache eviction time. This induces a short horizon bias under a limited cache budget : tokens with low current importance may be discarded before becoming critical for scene structure, object identity, or motion continuity, causing attention errors that accumulate into severe long horizon degradation. In this work, we identify an intriguing property of trained AR video models by examining query representations before and after RoPE modulation. In transformer attention, rotary positional embedding (RoPE) Su et al. 2021 is applied to queries before attention weights are computed, producing position encoded queries that determine how each frame attends to cached history. Our key observation is that these RoPE-modulated attention queries vary substantially across autoregressive steps, whereas their underlying pre-RoPE query distributions remain stable under a fixed prompt . This reveals a separation between stable semantic query structure and step dependent positional modulation, providing a basis for anticipating future attention behavior without additional training. Building on this insight, we propose Future Forcing, a future-aware training-free KV cache policy for AR video generation that preserves generation quality and consistency under a fixed cache budget to alleviate short-horizon limitations. Specifically, Future Forcing first constructs query proxies based on the stable pre-RoPE query distribution and uses them to score cached tokens by their predicted future importance, rather than relying only on their current or historical importance. Then, our Future Forcing further merges evicted tokens into retained entries with similar future-query-induced attention profiles to reduce compression-induced loss. In summary, our core contributions are as follows: Figure 1 : Qualitative comparison of 30s long video generation across different KV cache strategies. Competing methods exhibit temporal and subject inconsistency as generation progresses, while our Future Forcing maintains coherent subject consistency throughout the full AR generation process. • We investigate query representation dynamics in well-trained AR video generation models and uncover an intriguing phenomenon: RoPE-modulated queries vary substantially across autoregressive steps, whereas the underlying pre-RoPE query distribution remains stable under a fixed prompt. • We investigate the short-horizon limitation of existing KV cache compression methods, which rely on local context-derived signals and may discard tokens that are critical for future decoding. To address this, we propose Future Forcing, a future-aware training-free KV cache policy that uses future query proxies to guide token eviction and merging under the fixed KV cache budget. • Extensive experiments on representative autoregressive video generation models show that Future Forcing outperforms existing autoregressive video KV cache policies, achieving up to 1.49 subject-consistency improvement while maintaining generation quality under limited KV cache budgets. 2 Related Work 2.1 Bidirectional Video Generation Models. Recent progress in video generation has been largely driven by bidirectional diffusion-transformer models that jointly denoise entire video clips in spatiotemporal latent space. Sora Brooks et al. 2024 showed that scaling DiT pre-training on spacetime patches yields high-fidelity and temporally coherent videos, inspiring many follow-up systems. Open-source models including CogVideoX Yang et al. 2025b ; Hong et al. 2023 , HunyuanVideo Kong et al. 2024 ; Wu et al. 2025a ; Chen et al. 2025 , Wan Wan et al. 2025 , Open-Sora Zheng et al. 2024 , Latte Ma et al. 2025b , LTX-Video HaCohen et al. 2024 , Mochi-1 Team 2024 , Step-Video-T2V Ma et al. 2025a , and Pyramid Flow Jin et al. 2025 have substantially improved visual fidelity, motion quality, and text-video alignment. Proprietary systems such as Movie Gen Polyak et al. 2024 , Veo DeepMind 2024 , and Kling Team et al. 2025 follow similar design principles at even larger scales. However , jointly denoising all spatiotemporal tokens before decoding any frame precludes streaming generation during inference, making these methods ill-suited for real-time, low-latency interactive applications. 2.2 Autoregressive Video Generation Models. Autoregressive video generation models synthesize videos sequentially from previously generated context, making them naturally suited to streaming and long-horizon extension. Early works like VideoGPT Yan et al. 2021 , Phenaki Villegas et al. 2023 , VideoPoet Kondratyuk et al. 2024 , and Genie Bruce et al. 2024 established token-based and variable-length