Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation
Jiayu Chen, Junbei Tang, Wenbiao Zhao, Maoliang Li, Jiayi Luo, Zihao Zheng, Jiawei Yang, Guojie Luo, Xiang Chen · 2026
Read Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation by Jiayu Chen, Junbei Tang, Wenbiao Zhao, Maoliang Li, Jiayi Luo, Zihao Zheng, Jiawei Yang, Guojie Luo, Xiang Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13111.txt.
Introduction
Chen and eight co-authors study long video generation, where accumulated errors degrade quality over time. Existing KV-cache strategies assume all attention heads need the same history. But the authors identify three head types: Anchor Heads needing broad long-range context, Wave Heads with periodic temporal dependencies, and Veil Heads focused on initial and adjacent frames. Their Pyramid Forcing assigns each head type its own cache policy with heterogeneous cache lengths. On VBench-Long it lifts the 60-second Self Forcing score from 77.87 to 81.21.
Catalog record
| CREATOR | Jiayu Chen, Junbei Tang, Wenbiao Zhao, Maoliang Li, Jiayi Luo, Zihao Zheng, Jiawei Yang, Guojie Luo, Xiang Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52793 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generation Jiayu Chen, Junbei Tang, Wenbiao Zhao, Maoliang Li, Jiayi Luo, Zihao Zheng, Jiawei Yang, Guojie Luo, Xiang Chen Abstract Autoregressive video generation enables streaming and open-ended long video synthesis, but still suffers from long-term degradation caused by accumulated errors. Existing KVCache strategies usually apply unified historical-frame retention, implicitly assuming homogeneous historical dependencies across attention heads. We revisit historical-frame attention and reveal three distinct head types: Anchor Heads require broad long-range context, Wave Heads exhibit periodic temporal dependencies, and Veil Heads focus on initial and adjacent frames. Based on this finding, we propose Pyramid Forcing , a head-aware pyramidal KVCache framework that identifies head types offline, assigns behavior-specific cache policies, and supports heterogeneous cache lengths via efficient ragged-cache attention. Experiments on Self Forcing and Causal Forcing show that Pyramid Forcing consistently improves long-horizon generation quality on VBench-Long, increasing the 60-second Self Forcing score from 77.87 to 81.21 while enhancing motion dynamics, visual fidelity, and semantic consistency. Project: https://if-lab-pku.github.io/Pyramid-Forcing/ . Figure 1 : Pyramid Forcing mitigates long-video degradation, including appearance drift and subject inconsistency, achieving a state-of-the-art VBench-Long score of 81.21 among all baselines. Figure 2 : Comparison of KVCache policies. Unlike Self Forcing and Deep Forcing with unified cache compression, Pyramid Forcing assigns head-aware strategies to Anchor, Wave, and Veil Heads for differentiated historical information allocation. 1 Introduction Autoregressive (AR) video generation models ( huang2025self ; teng2025magi ; zhu2026causal ) provide a natural pathway for long video generation by iteratively denoising future frames conditioned on previously generated frames. Unlike diffusion-based video generation models ( peebles2023scalable ; wan2025wan ; yang2024cogvideox ) constrained by a fixed temporal scope, AR video generation supports streaming generation, continuation and open-ended generation, making it promising for applications such as interactive generation ( he2025matrix ; mao2025yume ) , world simulation ( yuan2026inference ; xie2026generated ) , and embodied intelligence ( feng2025vidarc ; ye2026world ) . However, in practical scenarios, long video generation still suffers from long-term degradation. As illustrated in Fig. 1 , as the generation process progresses, visual appearance tends to drift, motion dynamics weaken, and object identity gradually becomes blurred. Existing works ( cui2025self ) have suggested that this degradation primarily arises from error accumulation during autoregressive generation, especially when full historical information is retained. To address this issue, prior methods attempt to mitigate error accumulation by retaining recent frames ( yang2025longlive ; liu2025rolling ) , preserving early target frames ( li2026rolling ; kim2026memrope ) , or selecting important historical information yi2025deep . However, these approaches remain limited in alleviating error accumulation. As shown in Fig. 2 , we further observe that most of them implicitly assume a uniform dependency pattern on historical information within the model, thereby adopting a unified coarse-grained cache compression strategy. To examine this assumption, we revisit the attention mechanism in autoregressive video generation models represented by Self Forcing, and analyze the dependency patterns and temporal behaviors of different attention heads with respect to historical information during long-term generation. Our analysis reveals significant heterogeneity in how attention heads utilize historical information: One group of heads maintains strong and stable attention over a wide range of historical frames, exhibiting long-range anchoring capability, which we term Anchor Heads . Another group shows periodic temporal fluctuations in attention over historical frames, demonstrating dynamic and oscillatory temporal preferences, which we term Wave Heads . A small subset of heads exhibits weak attention to recent history but is highly sensitive to the initial frame, where excessive historical input can even suppress their focus on the first frame, which we term Veil Heads . These findings indicate that the implicit assumption of uniform compression in existing methods does not hold. Different attention heads exhibit distinct patterns of dependency on historical information. Therefore, historical information should be allocated in a differentiated manner based on head-specific temporal behaviors to more effectively preserve critical context and suppress error accumulation. Based on this observation, we propose Pyramid Forcing, a head-aware pyramidal KVCache framework for long video generation, as shown in Fig. 2 (c). Specifically, we first introduce Offline Tri-Pattern Head Classification , which uses sign-rate statistics and frequency-domain periodicity to identify Anchor, Wave, and Veil Heads from a small calibration set. Then, we introduce head-specific Pyramid KVCache Policies tailored to distinct historical-frame attention patterns, using strided retention for Anchor Heads, period-aligned sampling for Wave Heads, and compact local merging for Veil Heads to balance long-range coverage, periodic dependency, and local continuity. We further introduce Heterogeneous KVCache Computation Optimization , enabling low-overhead ragged-cache attention for variable head-level cache lengths. We evaluate Pyramid Forcing on Self Forcing and Causal Forcing using VBench-Long over 30s and 60s generations. Pyramid Forcing consistently improves long-horizon quality across both backbones, increasing the 60s total score of Self Forcing from 77.87 to 81.21. It also yields substantial gains in dynamic degree, with improvements of up to 29.36 points on Causal Forcing, demonstrating stronger motion dynamics, visual fidelity, and semantic consistency in long video generation. 2 Preliminaries 2.1 Autoregressive Video Diffusion Autoregressive video diffusion models generate videos frame by frame during the denoising diffusion process, conditioned on previously generated frames. Given a video sequence of N frames, its joint probability distribution can be factorized in an autoregressive manner as p(x^{1:N})=\prod_{t=1}^{N}p(x^{t}\mid x^{<t}) . Specifically, the previously generated frames x^{<t}=(x^{1},x^{2},\ldots,x^{t-1}) are modeled by the self-attention mechanism and used to predict the current frame x^{t} . The query of the current frame, Q\in\mathbb{R}^{B\times H\times L_{q}\times D} , attends to the keys and values of historical frames, K,V\in\mathbb{R}^{B\times H\times L_{kv}\times D} . The attention operation is computed as \operatorname{Attn}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\top}}{\sqrt{D}}\right)V. (1) Here, H denotes the number of attention heads, and D=C/H is the channel dimension of each head. L_{q} and L_{kv} denote the sequence lengths of the query and key/value tokens, respectively. The resulting attention logits typically have shape QK^{\top}\in\mathbb{R}^{B\times H\times L_{q}\times L_{kv}}. 2.2 KVCache Policy in Autoregressive Video Diffusion During autoregressive video generation, historical frames are encoded into latent tokens and stored as KVCaches. To reduce computation and mitigate error accumulation, existing methods maintain a rolling cache by feeding only selected historical frames into the model. When generating the t -th frame, a subset \mathcal{I}(t)\subset\mathcal{H}(t) is selected from the candidate history, typically consisting of sink frames \mathcal{S}(t) for global anchoring and recent frames \mathcal{R}(t) for local temporal context. Their KVCaches are concatenated as K^{\prime}=[K_{\text{sink}}\mid K_{\text{recent}}],\quad V^{\prime}=[V_{\text{sink}}\mid V_{\text{recent}}], and attention is computed by \operatorname{Attn}(Q_{t},K^{\prime}_{t},V^{\prime}_{t})=\operatorname{softmax}\left(\frac{Q_{t}[K_{\text{sink}}^{\top},K_{\text{recent}}^{\top}]}{\sqrt{D}}\right)[V_{\text{sink}},V_{\text{recent}}]. (2) After each generation step, the new keys and values are written into \mathcal{H}(t+1) and used to update \mathcal{I}(t+1) . Existing methods usually apply a unified cache policy across attention heads; in contrast, we analyze head-specific historical dependencies and optimize KV caching at a finer granularity. Figure 3 : Historical-frame attention patterns of three head types in Self Forcing. Anchor Heads capture broad long-range dependencies, Wave Heads exhibit periodic temporal selectivity, and Veil Heads focus on the first and adjacent frames while suppressing mid-history. Figure 4 : Analysis of head-wise periodicity and historical information demands. (a) Wave Heads exhibit a small and stable fluctuation period under FFT analysis, whereas others lack stable periodicity. (b) Different head types prefer distinct recent history, motivating a head-aware KVCache policy. 3 Motivation Autoregressive video generation predicts future frames by attending to the KVCache