In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
Taiying Peng, Jiacheng Hua, Miao Liu, Feng Lu · 2025
Read In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting by Taiying Peng, Jiacheng Hua, Miao Liu, Feng Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07447.txt.
Description
Taiying Peng and 3 co-authors benchmark how well AI models read user intent from first-person video in a 2025 paper. Egocentric video, shot from the wearer's viewpoint, records focus, actions, and context together, which could power proactive personal assistants. The team argues that gaze, where the eyes point, is a strong clue to intent that earlier benchmarks ignored.
Their benchmark, EgoGazeVQA, pairs gaze data with video questions generated by multimodal models and refined by human annotators. Existing models struggle to infer intentions accurately on the longer daily-life clips. Prompting methods that feed the model spatial, temporal, and intent cues drawn from gaze improve performance markedly. Further tests cover gaze-based fine-tuning and how errors in gaze estimation affect the gains. The results make a case for gaze as core input for personal AI.
Catalog record
| CREATOR | Taiying Peng, Jiacheng Hua, Miao Liu, Feng Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07447