Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs

Gengyuan Zhang, Mingcong Ding, Tong Liu, Yao Zhang, Volker Tresp · 2025

Read Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs by Gengyuan Zhang, Mingcong Ding, Tong Liu, Yao Zhang, Volker Tresp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-15457.txt.

Introduction

Gengyuan Zhang and four co-authors study how video models use memory of past scenes in a 2025 paper. Modern multimodal models grasp whole videos well, but streaming video, where events arrive one by one, is less explored. Remembering earlier events should help a model read the current one.

The authors confirm that memory helps. Yet memory built from the model's own past guesses can carry errors, and the model may then confabulate, inventing details to fit the false memory. Performance drops when that happens. They propose a fix that reviews and corrects stored memories before use. The paper frames memory as a double-edged tool for streaming video understanding.

Catalog record

CREATORGengyuan Zhang, Mingcong Ding, Tong Liu, Yao Zhang, Volker Tresp
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-15457

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs Gengyuan Zhang, Mingcong Ding, Tong Liu, Yao Zhang, Volker Tresp Abstract Multimodal large language models (MLLMs) have demonstrated strong performance in understanding videos holistically, yet their ability to process streaming videos—videos are treated as a sequence of visual events—remains underexplored. Intuitively, leveraging past events as memory can enrich contextual and temporal understanding of the current event. In this paper, we show that leveraging memories as contexts helps MLLMs better understand video events. However, because such memories rely on predictions of preceding events, they may contain misinformation, leading to confabulation and degraded performance. To address this, we propose a confabulation-aware memory modification method that mitigates confabulated memory for memory-enhanced event understanding. 1 Introduction Leveraging Multimodal Large Language Models (MLLMs) to understand events in videos ( Yu et al., 2024 ; Zhang et al., 2023 ; Maaz et al., 2024 ) has shown their effectiveness in a wide range of tasks due to the reasoning capabilities of LLMs. Much of the existing research focuses on understanding holistic videos ( Zhang et al., 2023 ; Li et al., 2024a ) , where videos are treated as a single entity and MLLMs process them in one go. In real-world scenarios, however, videos are often streaming and multi-event, and events unfold sequentially. Under such conditions, video events have significant temporal and contextual dependencies due to their inherent semantic relevance. Akin to human cognition, knowledge of past memories can help in understanding the present effectively. Figure 1: Knowing the memory of past events can help understand the current event. However, for streaming events in videos, we cannot access ground-truth narrations for previous events and this leads to confabulation. We thus begin by exploring how incorporating memory mechanisms in LLMs can help enhance event understanding in videos. A common practice for forming memory in LLMs is to prepend relevant contexts to the inputs of LLMs/MLLMs ( Behrouz et al., 2024 ; Zhang et al., 2024b ; Wang et al., 2024b ) where memories and knowledge are retained and recalled in tokens. Specifically, we specify the memory in the streaming video setting as event-triggered memory ( Fountas et al., 2025 ; Hatalis et al., 2023 ) since they are contextualized and dependent on the current scenes in videos, other than external event-agnostic knowledge. The memory consists of long-term memory ( Hatalis et al., 2023 ) , recollected from other episodes or videos with relevant events, and short-term memory of events that precede the current one. As shown in Fig. 1 and later in Tab. 1 , integrating memory as contexts, which consists of events and their narrations (detailed in Sec. 2 ), significantly outperforms the original 0-shot model, highlighting that knowing the past notably helps MLLMs. Nevertheless, this performance represents a gold-standard baseline—feasible only in ideal scenarios rather than in a practical streaming setting. In practice, MLLMs don’t access the ground truth memory, rendering the leveraged short-term memory prone to mispredictions in the previous rounds due to hallucinations or incomplete observations. We reevaluate the model under streaming conditions and observe a substantial performance drop. We attribute this performance gap primarily to misinformation generated in the memory, a phenomenon referred to as confabulation ( Sui et al., 2024 ) . To address the inevitable confabulated memory in the streaming setting, we introduce CAMEO, a confabulation-aware memory modification approach to mitigate the confabulation problem. Our paper can be summarized as follows: 1. We demonstrate that leveraging memory as context in MLLMs improves understanding of video events by introducing contextual and temporal knowledge. 2. Yet ground-truth memory is not accessible in the real streaming setting since the memory suffers from misinformation in MLLMs’ generation, namely confabulation; 3. We propose CAMEO, a memory modification approach, to mitigate confabulated memory in MLLMs. 2 Leverage Memory as Context in Event Understanding 2.1 Preliminary Given a video episode \mathcal{E} consisting of a sequence of N events \{\mathchoice{e_{1}}{e_{1}}{e_{1}}{e_{1}},\dots,\mathchoice{e_{N}}{e_{N}}{e_{N}}{e_{N}}\} with an arbitrary event e_{i} in the episode as a short video clip consisting of multiple frames. LLMs need to predict the narration \hat{t}_{n} of the current query event e_{n} with only access to the preceding events e_{1:n-1} . Memory Memory ( Fountas et al., 2025 ; Das et al., 2024 ) in LLMs, akin to human brains, refers to retrieving relevant events from past experiences. We denote the memory activated by the current event e_{n} as \mathcal{M} . We adopt two types of memory: long-term memory \mathcal{M}^{l} from the set of other episodes as persistent memory \mathcal{D} ; and short-term memory \mathcal{M}^{s} , which respectively means the memories that occurred earlier and memories in the current episode. Memory as contexts \mathcal{M} is defined as a set of events recollected: \displaystyle\mathcal{M} \displaystyle=\mathcal{M}^{l}\cup\mathcal{M}^{s}, (1) \displaystyle\mathcal{M}^{l} \displaystyle=\{(e^{l}_{\!1},t^{l}_{\!1}),\dots,(e^{l}_{\!N_{l}},t^{l}_{\!N_{l}})\}, \displaystyle\mathcal{M}^{s} \displaystyle=\{(e^{s}_{\!1},t^{s}_{\!1}),\dots,(e^{s}_{\!N_{s}},t^{s}_{\!N_{s}})\}. To collect events e^{l}_{\!j} in \mathcal{M}^{l} , we use a similarity-based retrieval method ( , ) . In contrast, for events e^{s}_{\!k} in \mathcal{M}^{s} , we rely on recency, selecting the most recent events relative to the current event (detailed in Sec. C.1 ). Memory as Context A prominent trend in memory-enhanced LLMs is to use memory as contexts in LLMs’ inputs ( Fountas et al., 2025 ; Behrouz et al., 2024 ) , thereby leveraging the In-Context Learning (ICL) capability of LLMs ( Brown et al., 2020 ; Gao et al., 2023 ) . We formulate long-term and short-term memories as prepended contexts to the inputs. Figure 2: Model Pipeline for memory as contexts for streaming events reasoning. We interleave the events and narrations from long-term and short-term memory as contextual inputs. Algorithm 1 Streaming Evaluation Input: Current query event e_{n} , Current episode up to current event \mathcal{E}_{:n-1} , Persistent memory \mathcal{D} Output: Prediction of narration t_{n} Initialize \mathcal{M}^{s}\leftarrow\emptyset ; for n=1 to N do Collect long-term memory from \mathcal{D} for k=1 to N_{l} do \mathcal{M}^{l}\leftarrow(e^{l}_{\!k},t_{k})\in\mathcal{D} ; end for \hat{t}_{n}\leftarrow\LLM(\mathcal{M},e_{n}) ; Update short-term memory with new prediction \mathcal{M}^{s}\leftarrow(e_{n},\hat{t}_{n}) ; end for return \hat{t}_{n} ; 2.2 Gain & Loss: Memory in Event Understanding An Upper Bound: Ground-truth Memory helps We begin by introducing a golden baseline: we use \mathcal{M} with ground-truth narrations in \mathcal{M}^{s} , denoted by \mathcal{M}^{s}_{gt} . In this setup, all memories are factual and plausible, therefore providing an upper bound on the performance of memory-enhanced LLMs. Confabulated Memory Misleads Streaming Event Understanding We next evaluate LLMs’ event understanding capabilities in a streaming scenario, as illustrated in Alg. 2 . This setting is more practical and challenging since the model cannot access ground-truth short-term memory \mathcal{M}^{s}_{gt} or rather the true narrations and can only utilize generated narrations up to the current event \mathcal{M}^{s}_{gen} . As shown in Fig. 2 , our streaming evaluation relies on short-term memory that incorporates earlier events and their predicted narrations. Each new event’s prediction is subsequently consolidated into the short-term memory of the current episode. 2.3 CAMEO: Mitigating Confabulation We propose a probing-and-surgical approach, CAMEO, C onfabulation- A ware ME mory m O dification, to mitigate confabulation in our setting. We treat confabulated memories as less factual and less credible contexts for LLMs. Accordingly, we propose to quantize the credibility of short-term memory and reduce the contribution of heavily confabulated memories. Uncertainty Estimation We adopt the semantic entropy proposed by Kuhn

Read the complete text