StreamGaze : Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal · 2025
Read StreamGaze : Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos by Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01707.txt.
Introduction
Daeun Lee and co-authors build StreamGaze for streaming video sense with gaze. Streaming tools process each frame as it comes in. They must also guess user intent for uses like AR glasses. Past streaming tests score time reasoning alone. None score use of human gaze.
Their system uses gaze signals to guide temporal thought. Gaze tells the model where the user cares. Proactive answers can then arrive before a full query. Gaze-guided streaming could make AR aids feel less like tools and more like attentive partners. Benchmark design is part of the work, since gaze use in streaming video had no shared test before this system set one out.
Catalog record
| CREATOR | Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01707
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StreamGaze : Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos Daeun Lee, Subhojyoti Mukherjee, Branislav Kveton, Ryan A. Rossi, Viet Dac Lai, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Mohit Bansal Abstract Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications like AR glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether MLLMs can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze , the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively evaluate streaming video understanding. These tasks assess whether models can use real-time gaze to follow shifting attention and infer user intentions from only past and currently observed frames. To build StreamGaze , we develop a gaze–video QA generation pipeline that aligns egocentric videos with raw gaze trajectories via fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that closely reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, revealing fundamental limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze-prompting strategies, reasoning behaviors, and task-specific failure modes, offering deeper insight into why current MLLMs struggle and what capabilities future models must develop. All data and code will be publicly released to support continued research in gaze-guided streaming video understanding. Figure 1 : StreamGaze ’s task taxonomy. We introduce StreamGaze , the first benchmark designed to evaluate how effectively MLLMs use gaze for temporal and proactive reasoning in streaming videos. We introduce gaze-guided past , present , and proactive streaming video understanding tasks. 1 Introduction Recent advances in Video Large Language Models (VideoLLMs) Lin et al. 2023 ; Zhang et al. 2023 ; Wang et al. 2025a have significantly improved multimodal understanding of dynamic visual environments. A growing line of work now investigates streaming video understanding Xiong et al. 2025 ; Chen et al. 2024 ; Fu et al. 2025b ; Zhang et al. 2024 , where models must process temporally incoming frames and respond in real time without access to future context. Such capability is essential for real-world applications such as robotics, embodied agents, and AR-glass assistants, where perception and decision-making must occur continuously as events unfold. Benchmarks play a critical role in tracking progress in this rapidly developing domain. While several streaming benchmarks exist Chen et al. 2024 ; Fu et al. 2025b ; Wang et al. 2025b ; Yang et al. 2025b ; Zhang et al. 2025 , they capture only some challenges faced by realistic streaming agents. As summarized in Table 1 , some benchmarks Yang et al. 2025b ; Xiong et al. 2025 focus primarily on past or present recognition, leaving out proactive understanding, the ability to anticipate future events and user intentions. Moreover, existing benchmarks rarely incorporate the human perceptual signals that actually drive real-world decision-making, even when they rely on egocentric video sources and implicitly assume AR-glasses usage scenarios. Among these perceptual cues, eye gaze is the most direct and reliable indicator of where a user is attending, what they are processing, and what they are likely to do next Zhang et al. 2017 ; Lai et al. 2024 ; Ilaslan et al. 2023 ; Li et al. 2021 . Omitting gaze removes the key perceptual signal humans use to filter relevant information, plan actions, and anticipate what will happen next. This leads to benchmark evaluations that diverge from the way humans actually perceive and reason about dynamic environments. However, incorporating gaze into video understanding has been challenging Peng et al. 2025 ; Ilaslan et al. 2023 . Unlike conventional QA that relies solely on visible content, gaze-guided QA requires interpreting dynamic gaze trajectories , modeling how attention shifts over time, and grounding these signals within a moving egocentric viewpoint. Raw gaze streams are noisy and contain fast, unstable fluctuations Stewart et al. 2020 ; Nuthmann and Canas-Bajo 2022 ; Kroell and Rolfs 2022 , while egocentric videos continuously shake and rotate with user motion, making even "what the user is looking at" a non-trivial spatio-temporal grounding problem. These challenges are further amplified in the streaming setting, where models must reason causally using only past and currently observed frames. Together, these factors make automatic construction of temporally grounded QA pairs highly challenging. To close this gap, we introduce StreamGaze , the first benchmark for gaze-guided streaming video understanding . As shown in Figure 1 , StreamGaze provides a unified suite of gaze-conditioned tasks spanning past , present , and proactive reasoning, enabling comprehensive evaluation of how well MLLMs use gaze for temporal understanding and future prediction. Solving these tasks requires models to interpret gaze online, follow shifting attention over time, and infer intention using only past and currently observed frames, mirroring the constraints faced by real streaming agents. To construct StreamGaze , we develop a semi-automatic gaze-guided data generation pipeline that aligns gaze trajectories with egocentric videos and converts them into temporally grounded QA pairs. We first detect stable gaze moments ( i.e . , fixations) throughout each video and extract objects within the corresponding gaze regions using region-specific visual prompting. Next, we capture the temporal dynamics of gaze ( i.e . , scanpaths) to model how a user’s attention shifts across spatial regions over time. Building on these signals, we automatically generate past, present, and proactive streaming tasks using an LLM, followed by human verification for quality assurance. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs (e.g., GPT-4o, InternVL-3.5) and both human performance and specialized streaming video understanding models. In particular, current MLLMs struggle to leverage gaze signals for temporal reasoning and proactive inference, revealing fundamental limitations in gaze-conditioned understanding. We further conduct detailed analyses of gaze-prompting strategies, gaze-based reasoning behaviors, and task-specific proactive responses on StreamGaze . Together, these evaluations position StreamGaze as the first comprehensive testbed for assessing gaze-driven causal reasoning and proactive prediction in streaming video scenarios. Dataset Avg video (s) QA pairs QA type Annotation Proac. Gaze All- time Ego. Gaze-based QA benchmarks GazeVQA Ilaslan et al. 2023 150 25040 MC Human ✘ ✓ ✘ ✓ EgoGazeVQA Peng et al. 2025 52 1757 MC Auto ✘ ✓ ✘ ✓ Streaming QA benchmarks StreamingBench Lin et al. 2024 268 4500 MC+OE Auto & Human ✓ ✘ ✓ ✘ SVBench Yang et al. 2025b 147 7374 OE Auto & Human ✘ ✘ ✘ \triangle OVO-Bench Niu et al. 2025 428 2814 MC+OE Auto & Human ✓ ✘ ✓ \triangle OmniMMI Wang et al. 2025b 324 2290 OE Auto & Human ✓ ✘ ✘ \triangle ProAssist Zhang et al. 2025 710 1020 OE Auto ✓ ✘ ✘ ✓ StreamBench Xiong et al. 2025 270 1800 OE Auto & Human ✘ ✘ ✘ ✓ Vispeak-Bench Fu et al. 2025b 21 1000 OE Auto & Human ✓ ✘ ✘ ✘ StreamGaze (Ours) 815 8521 MC+OE Auto & Human ✓ ✓ ✓ ✓ Table 1 : Comparison of streaming video understanding benchmarks with StreamGaze . MC: Multiple-choice; OE: Open-ended; All-time: Covers past, present,