Episodic Memory Representation for Long-form Video Understanding

Yun Wang, Long Zhang, Jingren Liu, Jiaqi Yan, Zhanjie Zhang, Jiahao Zheng, Xun Yang, Dapeng Wu, Xiangyu Chen, Xuelong Li · 2025

Read Episodic Memory Representation for Long-form Video Understanding by Yun Wang, Long Zhang, Jingren Liu, Jiaqi Yan, Zhanjie Zhang, Jiahao Zheng, Xun Yang, Dapeng Wu, Xiangyu Chen, Xuelong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09486.txt.

Introduction

Yun Wang and a large team present Video-EM, a memory framework that helps video language models answer questions about long videos, in a 2025 paper. Models can only view a limited number of frames, and picking frames one by one yields repeats and broken story lines. Their method builds memories around events instead.

A language model agent first finds moments matching the question, then groups them into coherent events with time, place, and content noted. A self-check loop trims noise and adjusts event detail until the evidence suffices. The result is a compact event timeline that existing video models can read directly. No extra training or model changes are needed.

Catalog record

CREATORYun Wang, Long Zhang, Jingren Liu, Jiaqi Yan, Zhanjie Zhang, Jiahao Zheng, Xun Yang, Dapeng Wu, Xiangyu Chen, Xuelong Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-09486

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Episodic Memory Representation for Long-form Video Understanding Yun Wang, Long Zhang, Jingren Liu, Jiaqi Yan, Zhanjie Zhang, Jiahao Zheng, Xun Yang, Dapeng Wu, Xiangyu Chen, Xuelong Li Abstract Video Large Language Models (Video-LLMs) excel at general video understanding but struggle with long-form videos due to context-window limits. Consequently, recent approaches focus on keyframe retrieval, condensing lengthy videos into a small set of informative frames. Despite their practicality, these methods simplify the problem to static text-image matching, overlooking spatio-temporal relationships crucial for capturing scene transitions and contextual continuity, and may yield redundant keyframes with limited information, diluting salient cues essential for accurate video question answering. To address these limitations, we introduce Video-EM, a training-free framework inspired by the principles of human episodic memory, designed to facilitate robust and contextually grounded reasoning. Rather than treating keyframes as isolated visual entities, Video-EM explicitly models them as temporally ordered episodic events, capturing both spatial relationships and temporal dynamics necessary for accurately reconstructing the underlying narrative. Furthermore, the framework leverages chain-of-thought (CoT) thinking with LLMs to iteratively identify a minimal yet highly informative subset of episodic memories, enabling efficient and accurate question answering by Video-LLMs. Extensive evaluations on the Video-MME, EgoSchema, HourVideo, and LVBench benchmarks confirm the superiority of Video-EM, which achieves highly competitive results with performance gains of 4–9% over respective baselines while utilizing fewer frames. Figure 1: Illustration of the limitation of existing training - free sampling methods. (a) Treating frames as isolated instances results in temporal discontinuities that disrupt the semantic narrative of events. (b) Densely redundant frames with limited information can occupy the model’s limited context window, diluting key cues and compromising overall performance. Introduction The rapid advancement of Video Large Language Models (Video-LLMs) has achieved remarkable progress in video understanding ( Xu et al. 2021 ; Zhang et al. 2025a ) , particularly in video question answering ( Zhang, Li, and Bing 2023 ) , demonstrating strong potential for modeling real-world scenarios ( Song et al. 2025 ) . However, as video content moves from minutes to hours-long sequences, the limited context window of Video-LLMs poses a critical bottleneck for long-form video understanding. To bridge this gap, recent research on long-form video understanding has emphasized training-free frame sampling strategies ( Liu et al. 2025 ; Yu et al. 2024 ; Tang et al. 2025 ; Wang et al. 2025 ) . While effective, these methods typically simplify long-video understanding as static text–image matching by selecting query-relevant keyframes, resulting in two primary limitations. First, methods like ( Wang et al. 2025 ) process keyframes to per-frame captions in Video-LLMs, overlooking temporal dependencies essential for explicitly capturing scene transitions and contextual continuity in complex narratives. Second, query-driven sampling tends to yield visually redundant frames in long videos. Such redundancy may dilute the salient cues essential for accurate question answering, and increase computational costs while underutilizing the limited context window in Video-LLMs ( Sun et al. 2025 ) , as illustrated in Figure 1 . To mitigate these limitations, we introduce Video-EM, a training-free framework inspired by human cognitive mechanisms. Humans excel at lifelong reasoning by encoding salient experiences as hierarchical episodic memories 1 1 1 In humans, episodic memory refers to the capacity to form, store, and consciously retrieve specific past events within their unique spatio-temporal contexts, effectively enabling “mentally time travel” to reexperience those moments. , each anchored in specific spatio-temporal contexts ( Rothfuss et al. 2018 ; Hampton and Schwartz 2004 ) . Drawing on this insight, Video-EM reformulates long-form video question answering by treating isolated keyframes as temporally ordered key events. These events are transformed into episodic memory representations that capture both intricate spatial scene relationships and temporal dynamics, enabling robust and contextually grounded reasoning. Next, the framework leverages chain-of-thought (CoT) thinking with Large Language Models (LLMs) to evaluate whether the current episodic memory provides sufficient information for reliable video comprehension. Through this iterative reasoning process, the framework retrieves the minimal yet highly informative subset of episodic memories for subsequent answering by Video-LLMs, ensuring both efficiency and accuracy. Concretely, given a video-based query, Video-EM performs multi-grained semantic retrieval by matching CLIP embeddings of decomposed query tokens against frame embeddings. Rather than treating retrieved keyframes as isolated inputs, Video-EM orders them temporally as key events and applies adaptive event expansion to enhance semantic coherence, mitigating the risk of missing diverse contextual information essential for answering complex queries. It then encodes these expanded events into structured episodic memories by employing object detection and video captioning modules, extracting both scene relationships and dynamic event narratives. This allows the existing Video-LLMs to “replay” each moment with explicit grounding of when it happened, where it took place, what occurred, and which objects were involved. Finally, Video-EM employs a CoT strategy to iteratively reason over episodic memories and select the most relevant subset. For finer analysis, it recursively decomposes events into shorter temporal segments. This process yields a minimal yet highly informative memory set, enabling the downstream Video-LLM to answer the query accurately. In summary, our contributions are threefold: (1) We propose a novel paradigm for long-form video understanding as episodic memory retrieval and reasoning, moving beyond frame-centric methods to a human-inspired memory mechanism that captures spatio-temporal context for robust and contextually grounded reasoning. (2) We propose Video-EM, a training-free framework that integrates an episodic memory module with a chain-of-thought (CoT) thinking strategy. It efficiently retrieves a minimal yet informative subset of memories for Video-LLMs, enhancing both accuracy and efficiency. (3) Experiments on long video understanding benchmarks demonstrate that Video-EM outperforms prior methods and is broadly compatible with mainstream Video-LLMs, achieving consistent improvements. Related Work Long-form Video Understanding Long-video understanding is inherently more challenging than those faced in short-video or image-based tasks due to its complex temporal dynamics and significant visual redundancy ( Qian et al. 2024 ; Zeng et al. 2024 ) . While several methods, such as extended context windows ( Chen et al. 2024 ; Zhang et al. 2024a ; Team et al. 2023 ) and token compression ( Li, Wang, and Jia 2024 ; Weng et al. 2024 ; Song et al. 2023 ) , have attempted to address these issues, they often struggle to balance computational efficiency with performance, particularly when handling hours-long content. To mitigate these limitations, recent advances in query-aware keyframe sampling ( Tang et al. 2025 ; Yu et al. 2024 ; Liu et al. 2025 ) have been developed to aim to alleviate redundancy by selecting frames relevant to queries. Parallel efforts have extended Chain-of-Thought (CoT) thinking to Video-LLMs ( Fei et al. 2024 ; Zhang et al. 2023 ; Wang et al. 2024c ) , leveraging LLMs as agents to interpret visual content through enhanced reasoning capabilities. However, these approaches tend to process keyframes independently, lacking mechanisms to explicitly model temporal dependencies or evolving narratives. A representative work, VideoTree ( Wang et al. 2025 ) employs a CoT-based strategy to retrieve keyframes and generate frame-wise captions in isolation, but this approach overlooks the causal and temporal relationships necessary for coherent video understanding. To advance long-video understanding, we argue that it is essential to move beyond isolated frame processing toward temporally grounded reasoning. This requires capturing both fine-grained visual semantics and high-level narrative progression. By structuring

Read the complete text