ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory
Le Tu Ngoc Minh, Jinyeong Lim, Dongsu Han · 2026
Read ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory by Le Tu Ngoc Minh, Jinyeong Lim, Dongsu Han (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26762.txt.
Introduction
Le Tu Ngoc Minh, Jinyeong Lim, and Dongsu Han present ProtoKV, a memory system for understanding streaming video. Questions about a live stream can arrive late. A key cue may flash by long before anyone asks about it. Memory limits force the system to drop or dilute old content. Their design keeps the recent past exactly and summarises the distant past.
Older content is folded into a bank of prototypes with summary statistics. Each prototype acts like a compact stand-in token at query time. The total memory stays fixed no matter how long the stream runs. The approach fits standard attention machinery without changes. On streaming benchmarks, ProtoKV beat token-keeping rivals by up to twelve and a half points in the long-delay regime. Gains grew as the delay grew.
Catalog record
| CREATOR | Le Tu Ngoc Minh, Jinyeong Lim, Dongsu Han |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26762
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory Le Tu Ngoc Minh, Jinyeong Lim, Dongsu Han Abstract Streaming video understanding (SVU) must answer queries that arrive asynchronously while visual tokens stream continuously under strict GPU-memory and query-time latency budgets. A key challenge is delayed query: decisive cues may appear briefly, yet many subsequent updates occur before the query arrives, increasing the risk that those cues are evicted or diluted under bounded memory. We propose ProtoKV, a constant-footprint SVU memory that represents far history as a fixed-capacity summary state rather than retaining token instances. ProtoKV keeps an exact near-window KV cache and aggregates older content into a semantic–spatial prototype bank with residual statistics. At query time, each prototype is exposed through a bounded pseudo-token interface that is drop-in compatible with standard attention. Under matched budgets and comparable query-time cost, ProtoKV improves accuracy by up to 12.5 points over token-retention baselines on SVU benchmarks in the long-delay regime, with gains that grow as query delay increases. † † affiliationnotice: Equal contribution 1 Introduction Streaming video understanding (SVU) systems must reason over continuous video signals in real-time ( 30 ; 18 ; 16 ) . Despite progress on offline benchmarks, real-world deployment requires handling continuous frames and asynchronous queries under a fixed GPU memory budget ( 3 ) . In this setting, the model cannot assume access to the full clip nor condition computation on a known query; instead, it must maintain an evolving online state that remains immediately query-able upon demand ( 4 ; 30 ) . A central difficulty in SVU is delayed query, where decisive cues (e.g., a momentary state change) appear briefly and long before a query is issued ( 29 ; 8 ) . Here, delay is not only a time gap but also the amount of post-evidence update pressure accumulated before answering. As the temporal gap between evidence and query grows, answering quality depends increasingly on what the system preserves from far history; this induces a near–far regime where recent content is kept exactly within a short window, while older history must rely on a compact representation. Bounded-memory designs typically occupy two extremes: sliding-window baselines maintain constant memory by retaining only recent content but inevitably lose evidence once it expires ( 28 ; 2 ) . Conversely, offloading or retrieval systems recover fine-grained details but add significant query-time overhead and variance ( 5 ; 22 ) . Between these extremes lies a critical operating point for interactive systems: always-on; query-agnostic maintenance; a fixed on-device footprint usable directly at query time without reconstructing the past. Recent work advances this operating point via online KV-cache compression under a hard memory cap for SVU ( 10 ) . While this class of methods yields predictable query-time cost, online retention decisions are brittle when answers depend on rare but decisive evidence across long delays. More broadly, tying far-history to specific token instances means that once a token is dropped, its associated fine-grained cue is lost ( 31 ; 15 ) . We propose ProtoKV, a constant-footprint SVU memory mechanism in this regime that shifts far-history representation away from token retention via a two-tier online state: (i) a short near window storing exact KV for recent frames, and (ii) a fixed-capacity far memory summarizing older history into object-centric prototypes with lightweight residual statistics. ProtoKV does not retain token-level KV for the distant past; instead, it leverages pseudo-tokens and mass-aware weighting to allow standard attention to access far-history evidence under a constant query-time footprint. This design is tailored to delayed queries depending on past event occurrence rather than instantaneous visual content. Evaluations across four streaming video benchmarks demonstrate that ProtoKV improves accuracy and degrades more gracefully than token-retention baselines as evidence becomes increasingly distant under matched memory budgets. Our code is available at https://github.com/kaist-ina/ProtoKV . Contributions. (1) We introduce a far-memory mechanism for SVU that summarizes distant history into a fixed-capacity, spatiotemporally consistent prototype bank augmented with residual statistics. (2) We propose pseudo-token synthesis with mass-aware weighting, enabling standard attention to exploit far-history evidence under a bounded query-time footprint. (3) We introduce a delayed-query protocol with retrospective-query filtering to keep invariant ground truth. 2 Background and Problem Setup 2.1 KV Growth as an Operational Constraint in SVU Streaming video understanding (SVU) requires maintaining an online state to process continuous video streams and answer arbitrary queries without replaying past frames ( 4 ; 3 ) . In Transformer-based VLM, this state scales linearly with time as each visual token is appended to the KV cache ( 6 ) . Given the unbounded nature of streaming, this context growth inevitably exceeds fixed deployment budgets, creating a fundamental scalability bottleneck ( 11 ; 31 ) . This pressure is amplified in video due to the high token arrival rate, quickly producing large token histories even with aggressive sampling. As a result, SVU systems must maintain an online state that supports immediate answering at query time, but whose memory cost does not scale with stream length. While simple heuristics like window truncation or token downsampling can stabilize memory ( 28 ; 2 ; 14 ) , they do so by sacrificing long-range evidence or fine-grained details, respectively. Consequently, SVU demands bounded-memory mechanisms that preserve past representations in a fixed-capacity footprint while remaining compatible with standard attention at query time. 2.2 Bounded-Memory SVU and Prior Approaches In contrast to offline video understanding, where the full sequence is accessible and models can compress or revisit tokens with complete context ( 32 ; 7 ) , SVU must process frames online without knowing the stream horizon in advance. In practice, this makes it unrealistic to “buffer the whole video” before answering. We therefore focus on bounded-memory SVU under three deployment-driven constraints. First, the memory footprint must be capped by design (constant or hard-bounded). Second, the online state must be maintained query-agnostically: the system cannot tailor what it stores to a future query it has not seen yet, nor can it replay the stream at query time. Third, the stored state must be immediately consumable at query time without expensive reconstruction steps that introduce latency variance. While some works utilize end-to-end trained compression ( 3 ) , we focus on training-free, drop-in mechanisms that are easier to deploy across models and backbones. Prior approaches span a spectrum of trade-offs under these constraints. Sliding-window KV caching (SWA) guarantees constant memory by retaining only the most recent window, yielding predictable cost but losing evidence once it exits the window ( 28 ; 2 ) . Online token-retention/compression methods ( 10 ) maintain a fixed KV budget by selecting, merging, quantizing, or evicting token instances over time. While they can preserve some far history, they still represent far evidence as a subset of token instances, and must continually decide which instances survive under a hard cap. Offloading/retrieval-based SVU ( 5 ) stores history in slower tiers or retrieves it at query time to recover fine-grained details; however, the resulting query-time overhead and variance can be a practical weakness when responsiveness is central (e.g., storage/network contention or strict tail-latency targets). These trade-offs shape the operating point required for practical SVU deployments: always-on, query-agnostic maintenance of a constant-footprint on-device state that can be directly