CacheFlow: Compressive Streaming Memory for Efficient Long-Form Video Understanding

Shrenik Patel, Daivik Patel · 2025

Read CacheFlow: Compressive Streaming Memory for Efficient Long-Form Video Understanding by Shrenik Patel, Daivik Patel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13644.txt.

Introduction

Shrenik Patel and Daivik Patel present CacheFlow, a training-free pipeline that lets vision-language models answer questions over very long videos. Attention and key-value caches swell with runtime, forcing costly inference or short sliding windows that forget the past. CacheFlow instead prunes and compresses the stream as it plays.

A dynamic step drops patch tokens that barely change between frames, while surviving tokens are packed into fixed blocks. Each block's keys are summarized into a retrieval index and its full data stored aside, then only the most relevant blocks return at answer time. Tests on offline and streaming benchmarks beat strong baselines while processing far fewer tokens, with no fine-tuning needed.

Catalog record

CREATORShrenik Patel, Daivik Patel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13644

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CacheFlow: Compressive Streaming Memory for Efficient Long-Form Video Understanding Shrenik Patel, Daivik Patel Abstract Long-form video question answering (VQA) overwhelms current vision-language models (VLMs) because attention and key-value (KV) caches grow with runtime, forcing either expensive inference or near-sighted sliding windows. We introduce CacheFlow, a training-free pipeline that pairs Dynamic Token Dropping (DTD) with a compressive long-term memory. DTD prunes per-patch tokens online via cosine similarity to the previous frame, and surviving tokens are packed into fixed-size blocks. This online, per-frame processing makes our approach fundamentally suited for live streaming VQA. As blocks are processed, each one’s keys are summarized by a tiny recurrent encoder to form a retrieval index, while the block’s full KV pairs are offloaded and later rehydrated for generation, preserving answer fidelity. At inference, a consensus-based retrieval mechanism retrieves only the Top-K most relevant blocks and attends over both the retrieved and local context for precise, long-range reasoning. CacheFlow is drop-in, architecture-agnostic, and requires no fine-tuning. Experiments on both offline and streaming VQA benchmarks demonstrate that CacheFlow outperforms current strong baselines, while processing up to 87% less tokens. Our dual approach enables VLMs to be both efficient and context-aware, paving the way for practical long-form video understanding. Figure 1 : CacheFlow system overview. Dynamic Token Dropping prunes redundant tokens via inter-frame similarity, and surviving tokens are packed into fixed-size blocks. Each block is summarized by a GRU-based compressive memory, while full key–value pairs are offloaded. During inference, consensus-first retrieval rehydrates only the Top- K relevant blocks for efficient long-range reasoning. 1 Introduction Long-form video understanding presents a fundamental challenge for modern vision-language models (VLMs). While recent multimodal architectures such as LLaVA and Qwen-VL 28 have achieved remarkable progress on short clips, their attention-based mechanisms scale quadratically, and for videos spanning minutes or hours the cumulative key–value (KV) cache quickly exceeds GPU memory, forcing models to either truncate context or perform expensive recomputation. This leaves a persistent gap between VLM efficiency and the temporal complexity of real-world video. Existing approaches to long-term modeling provide only partial solutions. Sliding-window and recurrent attention methods preserve computational tractability but inherently discard distant context once older tokens fall outside the window, limiting temporal reasoning to short horizons. Retrieval-augmented and memory-based systems increase the effective context length, yet they often accumulate redundant frame-level features and depend on uniform pooling or static compression that blur fine-grained temporal structure. These methods either underutilize memory by repeatedly storing similar information or overcompress it by collapsing temporally distinct events into coarse averages. Consequently, current designs fail to achieve both efficiency and fidelity—maintaining compact memory representations without sacrificing precise, temporally grounded recall. We introduce CacheFlow , a training-free framework that bridges this gap through a two-pronged design: Dynamic Token Dropping (DTD) and GRU-based Compressive Memory (GCM) . DTD reduces redundancy at the source by adaptively pruning per-patch tokens whose features remain similar across frames, yielding a sparse yet information-rich token stream. The GCM module complements this by compressing each block’s keys through a lightweight recurrent encoder, producing temporally-aware summaries that enable fast retrieval while preserving semantic integrity. During inference, a consensus-first retrieval strategy selectively rehydrates only the most relevant cached blocks, allowing the model to attend jointly to both recent and retrieved context. This design achieves long-range temporal reasoning without overwhelming the KV cache or requiring model retraining. CacheFlow operates online, requires no fine-tuning, and integrates seamlessly with existing VLMs. Across multiple long-form and streaming VQA benchmarks, it consistently improves both efficiency and accuracy, reducing KV cache size by up to 87% while maintaining state-of-the-art performance. With dynamic token pruning and compressive memory, CacheFlow establishes a scalable foundation for practical, streaming-capable vision-language systems. 2 Related Work 2.1 Efficient Transformers for Vision Transformers have become the dominant backbone for vision tasks, yet their quadratic attention cost poses major challenges for scalability. A broad line of work seeks to improve efficiency by constraining the attention pattern or reducing token count. Hierarchical and window-based variants such as Swin Transformer 18 and Longformer 3 limit pairwise interactions to local neighborhoods, achieving subquadratic complexity at the expense of long-range dependencies. Linearized approximations including Performer 6 and Linformer 24 instead reparameterize the attention kernel to achieve linear time, but often degrade accuracy on dense spatial reasoning tasks. Complementary strategies such as token merging or pruning—exemplified by ToMe 5 and H-ViT 11 —fuse redundant patches to accelerate inference while maintaining visual coverage. Although these advances substantially reduce compute for individual images or short clips, they do not address the growing key–value (KV) cache that accumulates when processing long video streams, where redundancy arises not only spatially but also temporally. 2.2 Long-Context Video Understanding Modeling extended temporal context in video understanding remains an open frontier. Conventional sliding-window and recurrent attention architectures 4 ; 19 ; 1 ; 10 maintain efficiency by retaining only recent tokens, but this truncation inherently limits long-range reasoning. Retrieval-augmented and persistent-memory systems 23 ; 16 ; 26 ; 27 attempt to bridge this gap by storing and reusing intermediate representations, yet often at the expense of redundancy or with retrieval mechanisms too coarse to capture nuanced temporal cues. ReKV 9 introduced retrieval over stored KV-caches for streaming video reasoning, but its block-level design still produces overlapping storage and reduced temporal precision. Flash-VStream 31 advanced efficiency further through a hierarchical flash memory with synopsis and detail layers, enabling real-time inference but emphasizing throughput over representational compression. CacheFlow unifies temporal sparsity and compressive memory within the transformer architecture, offering a new pathway toward efficient long-term video reasoning . Through adaptive token pruning and a recurrent compressive cache, it maintains fine-grained temporal fidelity while adhering to tight memory and latency budgets. 2.3 Temporal Redundancy in Videos Natural videos exhibit substantial temporal redundancy, where consecutive frames differ only in minor spatial details. Early work addressed this redundancy through heuristic frame sampling 25 ; 15 or keyframe selection 14 , reducing computation at the cost of temporal continuity. Subsequent methods introduced motion-based or feature-level filtering 33 ; 22 to discard visually similar regions across frames. More recently, TimeChat 29 proposed differential token dropping based on inter-frame cosine similarity, showing that many patch-level features remain static over time. While effective for streaming efficiency, it does not maintain long-term contextual coherence once dropped tokens are removed from memory. CacheFlow extends this idea by pairing online, per-frame token pruning with a compressive recurrent memory that stores compact summaries of past context, achieving temporal sparsity without losing access to long-range dependencies. 3 Method We present CacheFlow , a training-free pipeline for long-video understanding built around two ideas: (i) Dynamic Token Dropping (DTD) to eliminate frame-to-frame redundancy at the source and (ii) a GRU-based Compressive Memory (GCM) to compress, store, and selectively reload long-range context as needed for answering. Figure 1 provides an end-to-end visual overview of the CacheFlow architecture, illustrating how Dynamic Token Dropping, GRU-based Compressive Memory, and consensus-based retrieval interact during inference. 3.1 System Overview Let a video be a sequence of frames, which the vision tower maps into a set of patch features for each frame X_{t}=\{x_{t,p}\}_{p=1}^{P} . Our DTD module computes a binary “keep” mask m_{t,p}\in\{0,1\} for each patch by comparing its feature vector x_{t,p} to the corresponding patch in the prior frame, producing a pruned set

Read the complete text