From Gaze to Guidance: Interpreting and Adapting to Users’ Cognitive Needs with Multimodal Gaze-Aware AI Assistants DOI: XXXXXXX.XXXXXXX Conference: Symposium on User Interface Software and Technology; November 2–5, 2026; Detroit, Michigan ISBN: 978-1-4503-XXXX-X/2026/11 CCS: Human-centered computing Interactive systems and tools

Valdemar Danry, Javier Hernandez, Andrew Wilson, Pattie Maes, Judith Amores · 2026

Read From Gaze to Guidance: Interpreting and Adapting to Users’ Cognitive Needs with Multimodal Gaze-Aware AI Assistants DOI: XXXXXXX.XXXXXXX Conference: Symposium on User Interface Software and Technology; November 2–5, 2026; Detroit, Michigan ISBN: 978-1-4503-XXXX-X/2026/11 CCS: Human-centered computing Interactive systems and tools by Valdemar Danry, Javier Hernandez, Andrew Wilson, Pattie Maes, Judith Amores (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08062.txt.

Description

Danry and four colleagues build a gaze-grounded multimodal assistant that reads users' cognitive needs from eye motion. Current language-model helpers answer well but lack action context showing when a user struggles. It uses first-person video with gaze layers to spot likely hard points and target follow-up help. A study with 36 participants compares the gaze-aware version against a text-only version. The gaze-aware version rates as much more accurate and tailored in judging reading habits, lifts recall, and elicits fewer spoken words, for leaner chats.

Catalog record

CREATORValdemar Danry, Javier Hernandez, Andrew Wilson, Pattie Maes, Judith Amores
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74361
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08062

Browse the full library · About the Tower of Babel