Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion

Peiliang Cai Evelyn Zhang Jiacheng Liu Hao Lin Ruiqi Zhang Weile Mo, Yue Ma Shikang Zheng Jiehang Huang Dongrui Liu Linfeng Zhang · 2026

Read Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion by Peiliang Cai Evelyn Zhang Jiacheng Liu Hao Lin Ruiqi Zhang Weile Mo, Yue Ma Shikang Zheng Jiehang Huang Dongrui Liu Linfeng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18346.txt.

Introduction

Peiliang Cai and ten colleagues propose Focused Forcing, a training-free way to shrink the key-value cache in autoregressive video diffusion, where each new chunk of frames conditions on past frames. Instead of sharing one history choice across a chunk, it picks the most relevant and distinctive past frames per frame and gives larger budgets to more important attention heads. Reported in 2026, the result runs up to 1.48 times faster while improving visual quality and text alignment.

Catalog record

CREATORPeiliang Cai Evelyn Zhang Jiacheng Liu Hao Lin Ruiqi Zhang Weile Mo, Yue Ma Shikang Zheng Jiehang Huang Dongrui Liu Linfeng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50429
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18346

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion Peiliang Cai Evelyn Zhang Jiacheng Liu Hao Lin Ruiqi Zhang Weile Mo, Yue Ma Shikang Zheng Jiehang Huang Dongrui Liu Linfeng Zhang Abstract Recent advances in autoregressive video diffusion have enabled sequential and streaming video generation. However, long-horizon generation requires increasingly large KV caches, making efficient compression without sacrificing quality challenging. Existing methods mostly select historical frames based on attention scores, but their context decisions remain coarse. When multiple frames are generated in the same chunk, these methods often apply a shared history selection to the whole chunk, score historical frames solely by attention, and assign head-wise budgets either uniformly or by attention-pattern heuristics rather than explicit head-importance estimation. We show that frames within the same generated chunk can depend on distinct historical frames, that the same historical frame can receive different attention scores as its relative temporal distance to the current frames changes, and that masking different heads induces unequal generation degradation. Motivated by these findings, we propose Focused Forcing , a training-free KV selection method that focuses cached history along both generated-frame and head dimensions. For each generated frame, Focused Forcing preserves the most relevant and distinctive historical frames by combining attention scores with diversity scores of historical frames, while assigning larger budgets to heads with higher estimated importance. Across multiple autoregressive generation paradigms, Focused Forcing achieves up to 1.48 × \textbf{1.48}\times end-to-end acceleration without training, while improving visual quality and text alignment . Our code will be released on GitHub. Figure 1: Overview of Focused Forcing. (a) Existing methods use shared history selection, attention-only scoring, and coarse head budgeting, illustrated here with uniform budgets. (b) Focused Forcing uses per-frame history selection, content-aware scoring, and head-adaptive budgets. (c) Focused Forcing achieves up to 1.48 \times speedup across multiple paradigms while preserving quality. 1 Introduction Recent advances in video diffusion models [ 41 , 18 , 47 , 31 , 4 ] have greatly improved visual fidelity and motion coherence, making long-form, interactive, and streaming video generation increasingly practical. Unlike bidirectional paradigms that synthesize an entire clip jointly, autoregressive video diffusion [ 51 , 6 ] generates video sequentially, often in chunks, and naturally supports online rollout with KV caching [ 21 , 26 , 59 ] . However, long-horizon rollout requires maintaining an ever-growing cache of previously generated frames, making later attention operations increasingly expensive in memory and computation. To improve efficiency, existing methods typically restrict or compress the accessible history with sliding-window or rolling-KV designs [ 21 , 9 , 8 , 59 ] . Some preserve temporal continuity through structured cache rules such as attention sinks or cache re-caching [ 48 , 46 ] , while others select or prune cached frames by attention scores [ 48 ] . Recent work also explores head-specific cache strategies [ 16 ] . Despite these advances, as illustrated in Fig. 1 (a), existing methods still make coarse-grained context decisions in three aspects: 1) Shared Chunk-Level Selection , where all frames in a generated chunk share the same history selection; 2) Attention-Only Ranking , ranking historical frames mainly by attention scores; and 3) Coarse Head-Wise Budgeting , allocating head-wise budgets uniformly or heuristically without explicit head-importance estimation. These designs reduce cache overhead, but do not fully match the fine-grained context requirements of autoregressive video generation. Figure 2: Attention is generated-frame-dependent and relative-temporal-distance-sensitive. Rows denote query frames in the current generated chunk, and columns denote historical key frames. Attention varies across query frames and changes with relative temporal distance. Attention patterns in autoregressive video diffusion suggest that history selection should be performed at a finer granularity. As shown in Fig. 2 , frames within the same generated chunk can attend to different historical frames, indicating that context demand is generated-frame-dependent rather than shared by the whole chunk. Moreover, the same historical frame can receive different attention scores as its relative temporal distance to the current frames changes, suggesting that attention reflects not only content relevance but also temporal position. Therefore, selecting history with a single aggregated attention score may discard frames that are important for specific generated frames, while attention-only ranking can be unstable as relative temporal distance affects attention scores. Beyond selecting historical frames for each generated frame, fine-grained cache compression also requires allocating appropriate history budgets across attention heads. Existing head-specific strategies mainly rely on aggregated attention scores or heuristic attention patterns, but they do not explicitly estimate head importance for generation quality. As a result, they lack a direct criterion for head-wise KV budget allocation. This motivates assigning larger historical budgets to more important heads. Taken together, these observations motivate fine-grained KV compression across two dimensions: per-frame history selection and per-head budget allocation. Instead of pruning historical frames with a single aggregated attention score, KV compression should determine which historical frames each generated frame should access and how much history each attention head should retain. Meanwhile, the selection criterion should go beyond attention alone, since attention scores can be affected by relative temporal distance. Motivated by this formulation, we propose Focused Forcing , a training-free KV compression method for autoregressive video generation. As shown in Fig. 1 (b), Focused Forcing addresses fine-grained context allocation with three key designs: 1) Generated-Frame-Wise History Selection. Focused Forcing selects historical frames separately for each generated frame, rather than using one shared selection for the whole generated chunk. 2) Content-Aware Scoring. Focused Forcing combines attention relevance with diversity scores of historical frames, preserving distinctive historical frames when attention scores are affected by relative temporal position. 3) Head-Wise Budget Allocation. Focused Forcing estimates head importance by masking each attention head during denoising rollout and measuring the resulting degradation with a distribution-matching loss, then assigns larger KV budgets to more influential heads. Experiments across multiple autoregressive generation paradigms show that Focused Forcing consistently reduces cache overhead without additional training. As shown in Fig. 1 (c), it achieves up to 1.48 \times end-to-end acceleration while also improving visual quality and text alignment, demonstrating that fine-grained KV compression enhances both efficiency and quality in long-horizon video generation. Our contributions are summarized as follows: • We observe that attention-based history selection in autoregressive video generation is generated-frame-dependent and relative-position-sensitive, and that different attention heads have unequal impact on generation quality. • We propose Focused Forcing , a training-free KV compression method that performs fine-grained context allocation through generated-frame-wise history selection, content-aware scoring with attention and diversity scores of historical frames, and head-wise differentiated KV budgets. • Experiments across multiple autoregressive generation paradigms show that Focused Forcing achieves up to 1.48 \times end-to-end acceleration without training, while improving visual quality and text alignment . 2 Related Work Video Diffusion Models. Diffusion models [ 19 , 32 , 30 , 3 , 33 ] have set a strong baseline for image and video generation. Benefiting from scaled data and compute, recent video generators achieve high visual fidelity and temporal consistency, e.g., Wan [ 41 ] , HunyuanVideo 1.5 [ 42 ] , LTX Video [ 17 ] , and Seedance2 [ 36 ] , most of which rely on bidirectional spatiotemporal attention that ensures global consistency but is costly for long-form or interactive use. To support long-horizon and online

Read the complete text