Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation

Zengqun Zhao 1 , Yanzuo Lu 2 , Ziquan Liu 1 , Jifei Song 3 , Jiankang Deng 2 , Ioannis Patras 1 · 2026

Read Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation by Zengqun Zhao 1 , Yanzuo Lu 2 , Ziquan Liu 1 , Jifei Song 3 , Jiankang Deng 2 , Ioannis Patras 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21366.txt.

Introduction

Zhao and co-authors introduce Relax Forcing, a memory mechanism for long autoregressive video generation. Forcing based training reduces exposure bias, but minute scale videos still degrade over time. Their analysis shows that keeping more past frames does not always help. Relax Forcing splits temporal context into three roles: sink frames for global stability, tail frames for short term continuity, and selected history frames for mid-range motion. On VBench-Long, it improves motion dynamics and overall scores over autoregressive baselines.

Catalog record

CREATORZengqun Zhao 1 , Yanzuo Lu 2 , Ziquan Liu 1 , Jifei Song 3 , Jiankang Deng 2 , Ioannis Patras 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58010
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21366

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation Zengqun Zhao 1 , Yanzuo Lu 2 , Ziquan Liu 1 , Jifei Song 3 , Jiankang Deng 2 , Ioannis Patras 1 Abstract Autoregressive (AR) video diffusion has recently emerged as a promising paradigm for long video generation, enabling causal synthesis beyond the limits of bidirectional models. To address training–inference mismatch, a series of self-forcing strategies have been proposed to improve rollout stability by conditioning the model on its own predictions during training. While these approaches substantially mitigate exposure bias, extending generation to minute-scale horizons remains challenging due to progressive temporal degradation. In this work, we show that this limitation is not primarily caused by insufficient memory, but by how temporal memory is utilised during inference. Through empirical analysis, we find that increasing memory does not consistently improve long-horizon generation, and that the temporal placement of historical context significantly influences motion dynamics while leaving visual quality largely unchanged. These findings suggest that temporal memory should not be treated as a homogeneous buffer. Motivated by this insight, we introduce Relax Forcing, a structured temporal memory mechanism for AR diffusion. Instead of attending to the dense generated history, Relax Forcing decomposes temporal context into three functional roles: Sink for global stability, Tail for short-term continuity, and dynamically selected History for structural motion guidance, and selectively incorporates only the most relevant past information. This design mitigates error accumulation during extrapolation while preserving motion evolution. Experiments on VBench-Long demonstrate that Relax Forcing improves motion dynamics and overall temporal consistency while reducing attention overhead. Our results suggest that structured temporal memory is essential for scalable long video generation, complementing existing forcing-based training strategies. Figure 1 : Long-horizon autoregressive video generation over one minute. Baseline Self Forcing suffers from severe temporal drift and visual degradation as errors accumulate. Adding an attention sink stabilises early frames but limits motion diversity. Rolling-Forcing improves short-term coherence but still exhibits inconsistent patterns. In contrast, our Relax-Forcing dynamically selects informative history while suppressing redundancy, achieving stable identity preservation and sustained scene evolution. 1 Introduction Recent advances in video diffusion models have enabled high-fidelity synthesis of short videos [ 1 , 33 , 17 , 31 ] . To support interactive [ 32 , 26 , 38 ] and streaming [ 2 , 34 , 12 ] applications that require minute-scale generation, autoregressive (AR) diffusion models have emerged as a practical solution, generating frames causally with key–value (KV) caching mechanisms [ 3 , 29 , 37 , 13 , 21 ] . Rollout-based training strategies, such as Self Forcing [ 13 ] and its subsequent extensions [ 6 , 22 , 40 , 23 , 46 , 5 ] , mitigate exposure bias by allowing the model to condition on its own generated frames during training. During inference, these models extrapolate beyond the training horizon by autoregressively rolling out generation in a sliding window manner, where newly generated frames are appended to the context while older frames are discarded. This enables videos longer than the training sequence length to be synthesised. While rollout-based training substantially improves robustness, extending AR diffusion to minute-scale synthesis remains fundamentally challenging, a problem commonly referred to as long-video extrapolation [ 43 , 44 ] . Even with reduced training–inference mismatch, autoregressive models must repeatedly condition on previously generated frames, allowing residual errors and temporal biases to accumulate over time. Consequently, long-horizon generation often exhibits either gradual drift or overly constrained motion dynamics. Recent work therefore shifts attention from training strategies to memory management during inference [ 8 , 41 , 35 , 12 , 39 , 36 , 15 , 42 , 47 , 38 , 7 , 4 , 5 ] . For example, Deep Forcing [ 36 ] compresses candidate memories based on query-averaged attention scores, while Reward Forcing [ 22 ] preserves full historical information through an exponential moving average mechanism. Other approaches explore memory compression to summarise long video histories into compact contexts [ 42 , 47 ] . Despite these advances, existing studies largely treat memory as a capacity or update problem, leaving the temporal roles of different historical frames insufficiently understood. In particular, it remains unclear how much past information should be preserved and which parts of the history are most beneficial for long-horizon extrapolation. Figure 2 : Analysis of temporal memory in long-video extrapolation. (a-c): Varying the number of Sink, History, and Tail frames shows that increasing conditioning memory size does not consistently improve generation quality and may constrain motion dynamics. (d): Under a fixed memory budget, selecting history frames from different temporal positions leads to noticeable variations in motion dynamics while leaving image quality largely unchanged. Through empirical analysis of temporal memory configurations, we find that dense memory is not uniformly beneficial. Instead, its effectiveness depends critically on both the quantity and the temporal placement of stored frames. First, as shown in Fig. 2 (a-c), increasing the number of frames for conditioning, whether from Sink, middle-range History, or Tail, does not consistently improve long-horizon generation. Beyond a certain point, additional memory introduces redundancy that weakens motion dynamics without yielding meaningful gains in visual quality. Second, the temporal placement of historical frames also plays an important role. As shown in Fig. 2 (d), mid-range history contributes differently depending on where it is sampled, suggesting that not all past context is equally useful for future generation. Together, these findings indicate that temporal memory should not be treated merely as a chronological buffer whose effectiveness depends only on its size. Although existing approaches often employ memory compression or selection mechanisms, they primarily focus on improving scalability or retaining informative tokens, while the functional roles of different temporal regions remain underexplored. Simply retaining more past frames, or selecting historical context without considering their temporal roles, may even hinder long-horizon extrapolation. Instead, effective memory design must account for both how much past information is preserved and where it originates in the temporal sequence. This naturally calls for a structured view of temporal memory, where past frames play distinct functional roles rather than contributing uniformly. Figure 3 : Overview of Relaxed KV Memory. Instead of retaining dense chronological history, temporal memory is decomposed into three functional components: Sink for global anchors, History for intermediate motion structure, and Tail for recent continuity. During generation, candidate historical frames are dynamically selected to remain aligned with Sink while avoiding redundancy with Tail. The selected memory is then integrated through a relaxed KV formulation with adjusted relative positional encoding, enabling the model to leverage non-contiguous temporal context while preserving long-range consistency during autoregressive rollout. Motivated by this insight, we propose Relaxed KV Memory, a structured sparse memory mechanism for AR diffusion, as shown in Fig. 3 . Instead of employing the full or dense memory sequence as conditions, our approach decomposes temporal context into three functional roles: Sink for long-term stability, Tail for short-term continuity, and dynamically selected History for mid-range structure, and selectively incorporates only the most relevant historical frames during generation. In particular, History frames are selected using a relaxation-based scoring mechanism that balances global stability against redundancy with recent context. This enables the model to prioritise informative mid-range frames while suppressing overly correlated historical signals. Consequently,

Read the complete text