Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models

Tri Cao, Khoi Le, Thong Nguyen, Cong-Duy Nguyen, Quynh Vo, Anh-Tuan Luu, Miao Chunyan, See-Kiong Ng, Shuicheng Yan, Bryan Hooi · 2026

Read Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models by Tri Cao, Khoi Le, Thong Nguyen, Cong-Duy Nguyen, Quynh Vo, Anh-Tuan Luu, Miao Chunyan, See-Kiong Ng, Shuicheng Yan, Bryan Hooi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08974.txt.

Introduction

Tri Cao and nine coauthors diagnose why video language models hallucinate in moving scenes: they fail to keep track of objects, their states, and their relations over time. Final answer benchmarks hide the flaw because many questions yield to local cues or statistical habits. Their STEMO-Bench breaks queries into human verified object centered sub questions, separating genuine temporal understanding from lucky guesses.

Their companion framework, STEMO-Track, builds explicit object trajectories by extracting states chunk by chunk and aggregating them over time, then reasons over that structure. Across experiments it cuts hallucinated answers sharply and steadies spatio temporal reasoning against leading multimodal models. Video understanding gains memory with names and histories attached to things, not just frames.

Catalog record

CREATORTri Cao, Khoi Le, Thong Nguyen, Cong-Duy Nguyen, Quynh Vo, Anh-Tuan Luu, Miao Chunyan, See-Kiong Ng, Shuicheng Yan, Bryan Hooi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08974

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models Tri Cao, Khoi Le, Thong Nguyen, Cong-Duy Nguyen, Quynh Vo, Anh-Tuan Luu, Miao Chunyan, See-Kiong Ng, Shuicheng Yan, Bryan Hooi Abstract While multimodal large language models (MLLMs) have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. We argue this stems from a failure in spatio-temporal monitoring, the ability to persistently track object identities, states, and relations over time. Existing benchmarks obscure this deficit by relying on single final-answer evaluations for queries that can often be resolved via local visual cues or statistical priors. To rigorously diagnose this, we introduce STEMO-Bench ( S patio- TE mporal MO nitoring), a benchmark of human-verified object-centric facts that evaluates intermediate reasoning by decomposing queries into sub-questions, distinguishing genuine temporal understanding from coincidental correctness. To address failure modes exposed by STEMO, we propose STEMO-Track , a novel object-centric framework that explicitly constructs and reasons over structured object trajectories via chunk-wise state extraction and temporal aggregation. Extensive experiments demonstrate that our object-centric framework significantly reduces hallucinated answers and improves spatio-temporal reasoning consistency over state-of-the-art MLLMs. 1 1 1 Code: https://github.com/nguyentthong/video_hallucination † † ∗ Contributes equally, † Corresponding author. Figure 1: Comparison of existing evaluation benchmarks and MLLM architectures against our proposed STEMO-Bench and STEMO-Track. 1 Introduction Multimodal large language models (MLLMs) ( 32 ; 26 ; 37 ) have recently made rapid progress on video understanding, improving semantic reasoning ( 30 ) , instruction following ( 35 ) , and long context modeling ( 13 ; 28 ) . Despite these advances, their reliability remains limited, as MLLMs are still prone to hallucinations, producing plausible yet factually incorrect outputs ( 16 ; 7 ) . Understanding and diagnosing these failures is essential for deploying such systems in real world applications. At its core, video understanding involves reasoning over a dynamic world. A video consists of entities whose properties and relations evolve continuously over time. To faithfully interpret such content, a model must perform persistent spatio-temporal monitoring, including identifying objects, maintaining their identities, tracking their state changes, and linking them to temporally ordered events ( 34 ; 14 ) . This process requires consistent reasoning across time, where errors in early steps such as misidentifying an object or missing a state transition can propagate and lead to incorrect conclusions. However, existing evaluation benchmarks do not fully capture these requirements. Prior work has introduced a range of hallucination benchmarks ( 29 ; 12 ; 23 ) and mitigation strategies ( 31 ; 2 ; 25 ; 8 ) , typically by constructing fine-grained question-answer pairs using a combination of MLLMs and human annotators with access to raw video inputs such as frames and subtitles ( 29 ; 12 ) . While effective at exposing certain failure modes, these benchmarks exhibit two key limitations under the spatio-temporal monitoring perspective. First, many questions can be answered by inspecting a single frame or a short segment, without requiring consistent tracking of objects over time. This allows models to rely on local cues rather than performing global temporal reasoning. Second, evaluation is typically based on a single final answer ( 23 ; 4 ; 11 ) , which obscures the reasoning process. A model may arrive at the correct answer through statistical priors or partial observations, without correctly resolving the underlying spatio-temporal dependencies ( 22 ; 23 ; 11 ) . As a result, current benchmarks provide only a limited view of a model’s true understanding capabilities. To address this gap, we introduce STEMO-Bench ( S patio- TE mporal MO nitoring), a benchmark designed to explicitly evaluate an MLLM’s ability to track object-centric dynamics over time. STEMO-Bench adopts a structure-first data construction pipeline that aligns with the requirements of spatio-temporal monitoring. Instead of generating question-answer pairs directly from raw video inputs, we first construct a human-verified structural representation of each video, explicitly capturing objects, their evolving states, and temporal relations. These structured facts serve as the foundation for generating compositional question-answer pairs that require multi-step reasoning across time. By grounding all questions in explicit human-verified structure, STEMO-Bench ensures that each query directly probes temporal dependencies and object interactions, rather than superficial visual cues. Beyond dataset construction, STEMO-Bench introduces a fine-grained evaluation protocol that assesses the reasoning process itself. Each query is decomposed into a sequence of sub-questions corresponding to key steps in spatio-temporal monitoring, such as identifying objects, retrieving states at specific timestamps, and verifying temporal transitions. A model is considered consistent only if it correctly resolves these intermediate steps. This design allows us to distinguish genuine understanding from coincidental correctness, providing a more faithful evaluation of hallucination behavior. Motivated by the failure modes exposed by STEMO, we construct STEMO-Track , an object-centric pipeline for video understanding that explicitly models entities and their temporal evolution. Instead of treating a video as an unstructured sequence of frames, we represent it as a set of objects and their associated state trajectories, and perform reasoning over these structured representations. This perspective establishes a strong baseline, aligning the model’s internal computation with the requirements of spatio-temporal monitoring. Concretely, our pipeline first converts a video into a structured representation before performing question answering. We partition the video into temporal chunks and apply a state extractor to identify objects and their states over time, producing a set of local observations. A temporal aggregation module then links these observations across chunks, enforcing consistency over object identities, states, and timestamps to form coherent object trajectories. Given a question, the model retrieves the relevant trajectories and performs reasoning over them to produce the final answer. This design separates representation construction from query-specific reasoning, enabling more reliable modeling of long-horizon spatio-temporal dynamics. In summary, our contributions are threefold: (1) We introduce the STEMO-Bench alongside a faithful evaluation protocol that explicitly tests intermediate reasoning through atomic sub-questions. (2) We propose STEMO-Track a novel object-centric pipeline that systematically constructs and queries structured object trajectories from videos. (3) We conduct a systematic evaluation of modern MLLMs on target versus intermediate consistency, highlighting key spatio-temporal deficiencies and demonstrating the efficacy of explicit monitoring. 2 Related Work Video Multimodal Large Language Models. Recent years have seen rapid progress in multimodal large language models (MLLMs), evolving from general vision-language assistants into specialized systems capable of long-context temporal reasoning ( 32 ; 26 ) . Early models focused on instruction tuning and short-video understanding ( 30 ; 35 ) , while newer architectures emphasize long-context scaling, temporal grounding, and reasoning ( 13 ; 28 ; 37 ) . This advancement is driven by strong open-source/open-weight models such as InternVL3, Qwen3-VL, Qwen3.5-Omni, and Gemma-4 ( 36 ; 3 ; 21 ; 10 ) , alongside reasoning-focused video MLLMs including Cosmos-Reason2, VideoRFT, and Video-R1 ( 18 ; 27 ; 5 ) . Proprietary frontier models, including Gemini 3, GPT-5, and Claude-4.6-Sonnet, further continue to push multimodal reasoning capabilities ( 9 ; 20 ; 19 ; 1 ) . Video Hallucination Benchmarks and Mitigation. As video MLLMs advance, growing efforts focus on understanding and reducing their failure modes. Existing benchmarks target different vulnerabilities, including object hallucination ( 29 ) , action and temporal grounding errors ( 12 ) , structural reasoning deficits ( 23 ; 4 )

Read the complete text