Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
Baiyang Song 1 Yuli Lin 1 Qiong Wu 1 Tao Chen 1 Jun Peng 1 Xiao Chen 1 Yiyi Zhou 1† Rongrong Ji 1 1 Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China {songbaiyang, linyuli330, qiong, taochen}@stu.xmu.edu.cn [email protected] [email protected] {zhouyiyi, rrji}@xmu.edu.cn · 2026
Read Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding by Baiyang Song 1 Yuli Lin 1 Qiong Wu 1 Tao Chen 1 Jun Peng 1 Xiao Chen 1 Yiyi Zhou 1† Rongrong Ji 1 1 Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China {songbaiyang, linyuli330, qiong, taochen}@stu.xmu.edu.cn [email protected] [email protected] {zhouyiyi, rrji}@xmu.edu.cn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25658.txt.
Introduction
A video stream never ends, yet a model's memory must. Fixed budget streaming understanding is the problem Baiyang Song and colleagues at Xiamen University attack in 2026. Their method, CausalMem, needs no training. It keeps a bank of visual tokens. It judges which tokens repeat information already stored. Then it updates the bank through a running semantic basis that tracks the main themes seen so far. The idea mimics how a person summarises a long programme while watching it.
Applied to LLaVA-OneVision and Qwen2.5-VL, CausalMem beat prior methods on streaming and offline video tests alike. Average accuracy rose by a few percentage points. The compression was dramatic: about twelve thousand tokens held an hour of streaming video. That is a better than twentyfold squeeze, in some eighty two megabytes of storage. Careful forgetting, guided by redundancy, beat keeping everything.
Catalog record
| CREATOR | Baiyang Song 1 Yuli Lin 1 Qiong Wu 1 Tao Chen 1 Jun Peng 1 Xiao Chen 1 Yiyi Zhou 1† Rongrong Ji 1 1 Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China {songbaiyang, linyuli330, qiong, taochen}@stu.xmu.edu.cn [email protected] [email protected] {zhouyiyi, rrji}@xmu.edu.cn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25658
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding Baiyang Song 1 Yuli Lin 1 Qiong Wu 1 Tao Chen 1 Jun Peng 1 Xiao Chen 1 Yiyi Zhou 1† Rongrong Ji 1 1 Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China {songbaiyang, linyuli330, qiong, taochen}@stu.xmu.edu.cn [email protected] [email protected] {zhouyiyi, rrji}@xmu.edu.cn Abstract Currently, streaming video understanding is still a daunting task for existing multimodal large language models (MLLMs). Its difficulties not only lie in handling the ever-increasing video frames, but also in the unpredictability of future video content and input instructions. In this paper, we study this task from the perspective of constructing a dynamic but fixed-budget memory bank, and propose a novel and training-free approach termed CausalMem . CausalMem is dedicated to constructing a dynamic visual memory update mechanism, thereby maximizing the amount of information in streaming video within a limited memory space, much like the human brain. In practice, CausalMem estimates the redundancy of visual tokens and updates the memory bank via an online semantic basis, which models the principal semantics of the observed video stream. To validate CausalMem, we apply it to two representative MLLMs, namely LLaVA-OneVision and Qwen2.5-VL respectively, and conduct extensive experiments on both streaming and offline video understanding benchmarks. The experimental results not only show the great advantages than existing methods under both streaming and offline settings, e.g. , + 3.2 % +3.2\% and + 3.0 % +3.0\% average accuracy gains respectively, but also witness the superior semantic preservation for streaming videos, e.g. , using 12 k k token budgets to memorize hour-long streaming videos, which achieves more than 20 × \times visual token compression ratio and only occupies about 82 MB storage. Our code is given in CausalMem . † † footnotetext: † Corresponding author. 1 Introduction After achieving great successes in a variety of image-language tasks [ 1 , 7 , 32 , 24 ] , recent multimodal large language models (MLLMs) [ 34 , 23 , 20 ] also yield notable progress in the video domain [ 44 , 19 , 2 ] . To overcome the substantial computation caused by the excessive video tokens, numerous works [ 41 , 18 , 29 , 12 , 17 ] have been recently devoted to the research of efficient video understanding, employing techniques such as token pruning [ 38 , 41 , 18 ] and KV Cache compression [ 39 , 11 , 6 ] . In terms of video content compression, current research primarily focuses on offline video processing, which typically requires globally semantical compression strategies based on the entire video content. Moreover, for streaming video understanding, the fixed-budget compression still remains unexplored. To this end, efficient streaming video understanding is still an intractable problem for existing MLLMs [ 19 , 44 , 2 ] , of which difficulties mainly lie in two aspects. Firstly, MLLMs need to process the video of potentially unbounded length in the streaming scenario. For instance, LLaVA-OneVision [ 19 ] needs to cope with more than 700 k visual tokens for a two-hour video (1 FPS). Moreover, even the advanced video compression methods [ 5 , 18 ] will encounter a rapid accumulation of visual tokens over time, placing a heavy burden on both computation and storage. Secondly, when compressing streaming videos, it is almost impossible to predict future video content and potential user questions. This case renders many existing compression methods unapplicable, such as those based on global redundancy [ 18 , 29 ] or text-guided estimation [ 45 , 21 ] . Therefore, we raise a question that "Is it possible to achieve the dynamic and budgeted visual memory bank for efficient streaming video understanding, much like the human brain?" Figure 1 : (a) CausalMem consistently outperforms existing visual compression methods across streaming and offline video understanding benchmarks. The base MLLM used is LLaVA-OneVision. (b) Compared with existing streaming video methods, CausalMem maintains a fixed-size memory bank, enabling streaming video understanding with bounded computation and storage overhead. In this paper, we study this problem from the perspective of constructing a dynamic and fixed-budget visual memory bank, and propose a novel and training-free method for existing MLLMs, termed CausalMem . To relax the burdens of both computation and storage, CausalMem will maintain a fixed-size memory bank for streaming video tokens, while using a memory update mechanism to meet the properties of streaming video understanding. In practice, CausalMem draws on the principle of Information Maximization [ 10 ] and builds a compact but evolving semantic basis for the memory bank, thereby well estimating the visual tokens for memory update. In this way, CausalMem can not only well reduce the visual redundancy but also maximize the information gain of the memory bank. To validate CausalMem, we apply it to two MLLMs, namely LLaVA-OneVision [ 19 ] and Qwen2.5-VL [ 2 ] , and conduct extensive experiments on both streaming and offline video understanding benchmarks. The experimental results show that CausalMem not only achieves obvious performance gains than the compared streaming or offline methods, e.g. , +3.2\% and +3.0\% average performance gains on streaming and offline benchmarks respectively, but also preserve the principal semantics of streaming videos by only using 12 k token budgets, e.g. , achieving more than 20\times visual token compression ratio for an hour-long video while consuming the storage of only 82MB. Overall, our contributions are twofold: • We study the efficient streaming video understanding from the perspective of a dynamic yet fixed budget memory bank under strict causality, and propose a novel and training-free approach for existing MLLMs, termed CausalMem. • The proposed CausalMem obtains competitive or even better performance than existing efficient video compression methods on both streaming and offline benchmarks, and also exhibits its superiority in streaming video semantic compression with fixed token budgets. 2 Related Work Offline video understanding. Recent Multimodal Large Language Models [ 23 , 2 , 33 ] have shown strong capabilities in offline video understanding. However, as video length increases, the number of visual tokens increases linearly, leading to prohibitive computational cost and memory overhead. Existing methods address this issue from two main perspectives. One is adopting Retrieval-Augmented Generation (RAG), in which a subset of task-relevant frames is selected based on vision–text similarity, thereby reducing the number of frames processed by the MLLM. While effective, they rely on the availability of user queries in advance, limiting their applicability to online settings. The other one focuses on visual token compression. They reduce the token sequence length via token pruning or merging [ 18 , 29 , 3 , 28 ] , either based on global video content or query-aware relevance. Despite their effectiveness, they require whole video access and are thus unsuitable for streaming scenarios. Streaming Video Understanding. To address the above limitations, recent works focus on streaming video understanding, where visual tokens must be processed incrementally based on observed frames. Most existing methods aim to control the growth of visual tokens through memory or compression mechanisms. A representative direction is KV cache compression, where methods such as StreamMem [ 39 ] , ReKV [ 11 ] and LiveVLM [ 25 ] compress the key–value cache