VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Hoda Eldardiry, Pinar Yanardag · 2026
Read VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion by Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Hoda Eldardiry, Pinar Yanardag (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30351.txt.
Introduction
Hidir Yesiltepe and six co-authors present VideoMLA, a latent attention method for video diffusion. VideoMLA swaps per-head keys and values for one shared low-rank latent and one shared positional key, cutting per-token KV memory by about 93 percent. They ask why MLA works even though the spectrum idea used to back it in language models does not hold. They show the MLA bottleneck sets the true rank. On VBench, VideoMLA tops long-horizon scores and lifts speed by about a quarter on one B200.
Catalog record
| CREATOR | Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Hoda Eldardiry, Pinar Yanardag |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49905 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30351
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral, Adil Kaan Akan, Kaan Oktay, Hoda Eldardiry, Pinar Yanardag Abstract Long-rollout causal video diffusion has converged on a fixed-size sliding-window KV cache, with recent progress innovating within this layout by changing which tokens occupy the window or how their positions are encoded. The per-head KV layout itself, a dominant contributor to streaming memory and latency, has been mostly left unchanged. In this paper, we present the first study of Multi-Head Latent Attention (MLA) in video diffusion. VideoMLA replaces per-head keys and values with a shared low-rank content latent and a shared decoupled 3D-RoPE positional key, reducing per-token KV memory by 92.7 % 92.7\% at every cached layer. We further investigate why MLA succeeds in video diffusion even though the spectral assumption often used to motivate it in language models does not hold: pretrained video attention is not low-rank, with 99%-energy effective rank far above any practical latent dimension. VideoMLA retains quality at compression ratios where direct spectral approximation would predict large reconstruction error. We show that the MLA bottleneck, rather than the pretrained spectrum, determines the effective rank: both spectral and random initialization occupy nearly the full rank budget from initialization, and training preserves this budget while adapting within it. On VBench, VideoMLA matches short-horizon streaming video diffusion baselines, achieves the best overall score at long horizons among evaluated methods, and improves throughput by 1.23 × 1.23\times on a single B200. 1 Introduction Causal video diffusion models [ 9 , 4 , 16 , 31 , 25 , 24 , 17 , 27 , 22 , 3 , 23 , 12 , 30 , 13 ] have gained traction as the dominant approach to streaming, long-horizon video generation. Distilled from bidirectional teachers, they generate frames [ 31 , 5 , 11 , 8 ] or chunks autoregressively while attending to a rolling key-value (KV) cache of past frames, producing minute-long videos at interactive rates on a single GPU. As models scale toward longer rollouts, the per-head KV cache increasingly defines the operating point. At Wan-1.3B scale [ 21 ] , each cached token stores 2\times 12\times 128=3{,}072 dense KV scalars per layer, accounting for keys and values across 12 heads with 128 channels each. With a 21-latent-frame cache, 1,560 tokens per latent frame, and 30 transformer layers, the dense KV cache contains 3.02B scalars, or about 6.0GB in bf16/fp16. This footprint explains why recent streaming systems use fixed-size sliding-window caches: retaining all past KV states would grow linearly with rollout length. However, fixing the window only bounds the number of cached tokens; it does not reduce the per-token, per-layer cost of the per-head KV layout. Reducing this layout is therefore a direct lever for longer horizons, larger batches, and faster inference. The dominant line of recent work treats the cache as a fixed-size sliding window and innovates inside it. CausVid [ 27 ] initiated this thread by converting bidirectional diffusion into causal autoregressive generation via distribution matching distillation, with a sliding KV cache from inception. Self-Forcing [ 9 ] closed the train–test gap by conditioning training on self-generated frames within the same rolling cache. Subsequent work refined this recipe through attention-sink, token-selection, and compressed-memory mechanisms for long-range consistency [ 16 , 25 , 17 , 28 , 12 , 29 ] , training strategies for multi-minute rollouts and prompt switching [ 4 , 22 , 7 , 24 ] , improved distillation objectives [ 31 , 17 ] , and positional reparameterization such as Infinity-RoPE [ 24 ] . However, these methods all preserve the per-head KV layout that fills the window in the first place: they redistribute, reweight, compress over time, or reposition cached tokens without reducing the per-token KV state. A second, complementary line changes the attention computation itself. SANA-Video [ 3 ] replaces softmax attention with block-causal linear attention, removing the conventional KV cache and using a constant-memory cumulative state for long-video generation. SCD [ 2 ] reduces cached state by routing temporal reasoning through a 25-layer causal encoder and using a 10-layer frame-wise decoder, so only the encoder layers cache. Under the same Wan cache geometry, this reduces dense KV storage by 16.7\% . VideoMLA is orthogonal: it keeps all 30 self-attention layers cached but reduces each token’s cached state from 3072 to 224 scalars, yielding an 11.4\times smaller cache than SCD for the same 21-latent-frame window. Thus, rather than changing which tokens are cached, how they are positioned, or how many layers cache, VideoMLA targets the remaining factor directly: the per-token KV layout at every cached self-attention layer. Figure 1: Pretrained video diffusion attention is not low-rank, unlike in language models. Singular value analysis of [W_{K};\,W_{V}]\in\mathbb{R}^{3072\times 1536} across the 30 transformer blocks of Wan2.1-T2V-1.3B. At d_{c}=192 , the median layer captures only E_{\mathrm{med}}=0.458 of the spectral energy, and the 99%-energy effective rank exceeds 1300 in every layer. In this paper, we intervene on the per-head layout itself. Building on Multi-Head Latent Attention (MLA) [ 14 ] , we present VideoMLA, the first MLA-style latent KV cache for autoregressive video diffusion. VideoMLA replaces per-head keys and values with a shared low-rank content latent and a head-shared decoupled 3D-RoPE positional key, reducing per-token KV memory by 92.7\% at every cached layer. This raises a puzzle: MLA is usually motivated by low-rank pretrained W_{K},W_{V} [ 14 , 10 ] , yet Wan-1.3B (Fig. 1 ) has 99%-energy rank far above practical latent dimensions. VideoMLA nonetheless retains quality where direct spectral approximation would incur large reconstruction error (Fig. 2 ). We show that the MLA bottleneck, not the pretrained spectrum, determines the effective rank: SVD and random initialization both nearly saturate the rank budget, which training preserves with little spectral change. The design question therefore shifts from what is the intrinsic rank? to what latent budget preserves video quality? Our contributions are summarized as follows: • Latent KV caching for video diffusion. We introduce VideoMLA, an MLA-style autoregressive video diffusion model that replaces per-head keys and values with a shared content latent and a head-shared decoupled 3D-RoPE key, reducing per-token KV memory by 92.7\% at every cached layer. • A spectral puzzle and rank-budgeted resolution. We show that Wan-1.3B video attention is not low-rank: the 99%-energy effective rank of [W_{K};W_{V}] far exceeds practical latent dimensions. VideoMLA nevertheless retains quality, while both SVD and random initialization saturate the imposed rank budget from initialization and preserve it during training. • Efficient long-horizon generation. We identify the NoPE/RoPE allocation that preserves visual fidelity and motion consistency at minute-scale horizons. On VBench, VideoMLA matches short-horizon baselines, achieves the best long-horizon overall score among evaluated methods, and improves throughput by 1.23\times on a single B200. 2 Related Work Figure 2: The composed operator occupies its full rank- d_{c} budget at every d_{c} and every layer. Singular value analysis of the composed operator M_{\text{learned}}=[W^{K}_{\uparrow}W^{KV}_{\downarrow};W^{V}_{\uparrow}W^{KV}_{\downarrow}] for SVD-initialized VideoMLA students at d_{c}\in\{64,128,256,512\} . (a) Median normalized spectra share a common envelope, truncated at d_{c} . (b) Cumulative spectral energy. (c) Layer-wise 99%-energy effective rank: r_{0.99}\approx 0.98\,d_{c} at every budget, uniformly across depth. The composed operator’s rank is determined by the architectural bottleneck, not by the spectral structure of the