Eye Gaze as a Signal for Conveying User Attention in Contextual AI Systems Conference: 2025 Symposium on Eye Tracking Research and Applications; May 26–29, 2025; Tokyo, Japan 2025 Symposium on Eye Tracking Research and Applications (ETRA ’25), May 26–29, 2025, Tokyo, Japan DOI: 10.1145/3715669.37273

Ethan Wilson, Naveen Sendhilnathan, Charlie S. Burlingham, Yusuf Mansour, Robert Cavin, Sai Deep Tetali, Ajoy Savio Fernandes, Michael J. Proulx · 2025

Read Eye Gaze as a Signal for Conveying User Attention in Contextual AI Systems Conference: 2025 Symposium on Eye Tracking Research and Applications; May 26–29, 2025; Tokyo, Japan 2025 Symposium on Eye Tracking Research and Applications (ETRA ’25), May 26–29, 2025, Tokyo, Japan DOI: 10.1145/3715669.37273 by Ethan Wilson, Naveen Sendhilnathan, Charlie S. Burlingham, Yusuf Mansour, Robert Cavin, Sai Deep Tetali, Ajoy Savio Fernandes, Michael J. Proulx (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13878.txt.

Introduction

Ethan Wilson and colleagues proposed in 2025 that an AI assistant should watch where you look, not just listen to what you say. Multimodal agents increasingly collaborate with users on tasks in the physical and digital world. Today they depend on explicit communication: typed or spoken instructions. The authors hypothesize that gaze, an implicit signal, can convey a user's interests and intent continuously, without demanding attention to formulate commands.

Presented at an eye-tracking research symposium in Tokyo, the work explores how systems can consume gaze as an attention channel. When an assistant knows which object a user is considering, it can offer help earlier and ask fewer clarifying questions. The risks are real too, since gaze reveals more than users may intend. Framing gaze as a first-class input gives designers a shared problem to solve: reading attention helpfully without reading minds intrusively.

Catalog record

CREATOREthan Wilson, Naveen Sendhilnathan, Charlie S. Burlingham, Yusuf Mansour, Robert Cavin, Sai Deep Tetali, Ajoy Savio Fernandes, Michael J. Proulx
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-13878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Eye Gaze as a Signal for Conveying User Attention in Contextual AI Systems Conference: 2025 Symposium on Eye Tracking Research and Applications; May 26–29, 2025; Tokyo, Japan 2025 Symposium on Eye Tracking Research and Applications (ETRA ’25), May 26–29, 2025, Tokyo, Japan DOI: 10.1145/3715669.3727349 ISBN: 979-8-4007-1487-0/2025/05 CCS: Human-centered computing Natural language interfaces CCS: Human-centered computing Mixed / augmented reality CCS: Computing methodologies Spatial and physical reasoning Ethan Wilson, Naveen Sendhilnathan, Charlie S. Burlingham, Yusuf Mansour, Robert Cavin, Sai Deep Tetali, Ajoy Savio Fernandes, Michael J. Proulx Abstract . Advanced multimodal AI agents can now collaborate with users to solve challenges in the world. Yet, these emerging contextual AI systems rely on explicit communication channels between the user and system. We hypothesize that implicit communication of the user’s interests and intent would reduce friction and improve user experience when collaborating with AI agents. In this work, we explore the potential of wearable eye tracking to convey signals about user attention. We measure the eye tracking signal quality requirements to effectively map gaze traces to physical objects, then conduct experiments that provide visual scanpath history as additional context when querying vision language models. Our results show that eye tracking provides high value as a user attention signal and can convey important context about the user’s current task and interests, improving understanding of contextual AI agents. † † cc-license: by 1. Introduction Artificial intelligence (AI) agents have become more connected with users in daily life ( Wienrich and Latoschik 2021 ) , especially by observing context about the user’s prior actions or current world state ( Zhang et al. 2024a ) . New innovations, such as vision-language models (VLMs) ( Li et al. 2024 ) and machine perception devices ( Engel et al. 2023 ) , pave the way towards contextual AI agents: agents which “ see " the nearby physical world and collect / compile contextual cues to better assist users. Yet, current models interpret information differently than humans, so often misinterpret context, conflicting with user intent. If implicit information about the user’s state could be reliably supplied to the agent, the user and agent’s intent could be better aligned. Eye gaze has been hypothesized as a valuable signal for conveying intent to agents ( Ajanki et al. 2010 ; Büschel et al. 2018 ; Burlingham et al. 2024b ; Zhang et al. 2024b ) . Gaze conveys information about objects users are interested in, cognitive load, the current action, etc. ( Mahanama et al. 2022 ) , all of which could improve models’ understanding. While eye tracking (ET) is a common input in extended reality (XR) systems ( Plopski et al. 2022 ) , the use of ET in human-agent interactions has only been lightly explored ( Sendhilnathan et al. 2024 ) . ET signals could convey to an agent what the user is or has been interested in. Yet, wearable eye trackers are limited in accuracy due to multiple factors (system error, slippage, individual user differences, etc.) ( Ehinger et al. 2019 ) , constraining whether objects could be reliably identified. If an object’s visual size relative to the ET signal accuracy is too small, it could be unreliable to detect. We present an analysis of the requirements and benefits of ET in wearable contextual AI. Using a dataset of egocentric recordings taken during daily household tasks ( Pan et al. 2023 ) , we estimate the expected ET accuracy thresholds for detecting physical objects in different contexts. We then conduct a number of experiments where contextual information from ET is appended to VLM queries. These experiments reinforce ET’s value in this space, and show improvements in the model’s ability to perceive user attention and current actions. 1.1. Contributions (1) We estimate ET accuracy requirements needed for accurate gaze placement on physical objects, to determine ET accuracy requirements for wearable contextual AI and future systems. (2) We explore how ET information can be conveyed to AI agents, both as point-in-time and as scanpath information with temporal dependencies. By augmenting VLM queries by supplying ET context, we augment agentic ability to understand user attention and current actions. 1.2. Privacy and Ethics Statement Our findings convey ET’s usefulness in human-agent interactions. Eye movements are known to convey personal information and user preferences ( Bozkir et al. 2023 ) , so any contextual AI system incorporating ET must be secure and privacy-preserving to avoid revealing user characteristics to others. 2. Related Literature Eye tracking is being adopted heavily in XR, providing clear value in human-computer interaction (HCI) interfaces. As contextual AI emerges, new prototypes have explored eye gaze as a means to estimate user attention. This section provides an overview of related literature, including the use of ET for selection, scene understanding, and ET in contextual AI. 2.1. Eye Tracking for Selection in Extended Reality In recent years, eye gaze has gained popularity as a signal for HCI in XR systems ( Plopski et al. 2022 ) . Eye gaze has comparable usability to controllers ( Luro and Sundstedt 2019 ; Zhang et al. 2019 ; Fernandes et al. 2024 ) while freeing the hands for other tasks and being preferable to users ( Piening et al. 2021 ) . While ET is prone to a Midas touch fallacy, where false selections are made during ambient fixations ( Jacob 1995 ) , novel HCI methodologies ( Khamis et al. 2018 ) overcome this and make ET an ideal signal for interface navigation. Our work explores ET’s ability to continuously and implicitly convey a user’s attention and intent ( Sendhilnathan et al. 2024 ) . This has only lightly been explored with AI agents, though ET has facilitated automatic contextual displays in the past ( Toyama et al. 2012 ) . 2.2. Eye Gaze Encodes Scene Understanding Eye movements reflect a viewer’s internal processing of a scene, giving insights to cognitive state and attention as one interprets new visual stimuli ( Langton et al. 2000 ; Eckstein et al. 2017 ) . Sequences of gaze fixations (i.e., scanpaths) encodes contextual cues as to future objects of interest ( Itti and Koch 2001 ; Burlingham et al. 2024a ) ; a number of works have leveraged scanpath history for short-term gaze prediction / anticipation ( Huang et al. 2018 ; David-John et al. 2021 ; Hu et al. 2021 ; D’Amelio et al. 2024 ) . Burlingham et al. found temporal dependencies in scanpaths lasting for 4-5 fixations on average, with high variance across task types ( Burlingham et al. 2024b ) . Contextual AI models may be able to leverage the rich, multiscale structure of scanpaths when inferring intent. Insights about the cognitive encoding of nearby objects can inform our expectations for eye movements in contextual AI ( Tatler et al. 2011 ) . For example, humans tend to look at a coffee mug just before grasping, to encode the location of the handle so that it can be successfully grasped. Object locations are encoded via an egocentric reference to the user, and greater affordances are given to nearby interactable objects ( Costantini et al. 2010 ; Tatler et al. 2011 ) . The visual system elicits responses to reachable 3D objects, even when

Read the complete text