From Gaze to Guidance: Interpreting and Adapting to Users’ Cognitive Needs with Multimodal Gaze-Aware AI Assistants DOI: XXXXXXX.XXXXXXX Conference: Symposium on User Interface Software and Technology; November 2–5, 2026; Detroit, Michigan ISBN: 978-1-4503-XXXX-X/2026/11 CCS: Human-centered computing Interactive systems and tools
Valdemar Danry, Javier Hernandez, Andrew Wilson, Pattie Maes, Judith Amores · 2026
Read From Gaze to Guidance: Interpreting and Adapting to Users’ Cognitive Needs with Multimodal Gaze-Aware AI Assistants DOI: XXXXXXX.XXXXXXX Conference: Symposium on User Interface Software and Technology; November 2–5, 2026; Detroit, Michigan ISBN: 978-1-4503-XXXX-X/2026/11 CCS: Human-centered computing Interactive systems and tools by Valdemar Danry, Javier Hernandez, Andrew Wilson, Pattie Maes, Judith Amores (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08062.txt.
Description
Danry and four colleagues build a gaze-grounded multimodal assistant that reads users' cognitive needs from eye motion. Current language-model helpers answer well but lack action context showing when a user struggles. It uses first-person video with gaze layers to spot likely hard points and target follow-up help. A study with 36 participants compares the gaze-aware version against a text-only version. The gaze-aware version rates as much more accurate and tailored in judging reading habits, lifts recall, and elicits fewer spoken words, for leaner chats.
Catalog record
| CREATOR | Valdemar Danry, Javier Hernandez, Andrew Wilson, Pattie Maes, Judith Amores |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74361 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08062