In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting

Taiying Peng, Jiacheng Hua, Miao Liu, Feng Lu · 2025

Read In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting by Taiying Peng, Jiacheng Hua, Miao Liu, Feng Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07447.txt.

Description

Taiying Peng and 3 co-authors benchmark how well AI models read user intent from first-person video in a 2025 paper. Egocentric video, shot from the wearer's viewpoint, records focus, actions, and context together, which could power proactive personal assistants. The team argues that gaze, where the eyes point, is a strong clue to intent that earlier benchmarks ignored.

Their benchmark, EgoGazeVQA, pairs gaze data with video questions generated by multimodal models and refined by human annotators. Existing models struggle to infer intentions accurately on the longer daily-life clips. Prompting methods that feed the model spatial, temporal, and intent cues drawn from gaze improve performance markedly. Further tests cover gaze-based fine-tuning and how errors in gaze estimation affect the gains. The results make a case for gaze as core input for personal AI.

Catalog record

CREATORTaiying Peng, Jiacheng Hua, Miao Liu, Feng Lu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07447

Browse the full library · About the Tower of Babel