GazeSummary: Exploring Gaze as an Implicit Prompt for Personalization in Text-based LLM Tasks Conference: The 27th International Workshop on Mobile Computing Systems and Applications; February 25–26, 2026; Atlanta, GA, USA The 27th International Workshop on Mobile Computing Systems and Applications

Jiexin Ding, Yizhuo Zhang, Xinyun Liu, Ke Chen, Yuntao Wang, Shwetak Patel, Akshay Gadre · 2026

Read GazeSummary: Exploring Gaze as an Implicit Prompt for Personalization in Text-based LLM Tasks Conference: The 27th International Workshop on Mobile Computing Systems and Applications; February 25–26, 2026; Atlanta, GA, USA The 27th International Workshop on Mobile Computing Systems and Applications by Jiexin Ding, Yizhuo Zhang, Xinyun Liu, Ke Chen, Yuntao Wang, Shwetak Patel, Akshay Gadre (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17676.txt.

Introduction

Jiexin Ding and six co-authors present GazeSummary, a study of gaze as a prompt for assistants. Smart glasses now pair language models with rich inputs. Most inputs still need explicit prompts, which can be obtrusive. Gaze tracking offers an implicit signal of intent. The team asks if models can read gaze for text tasks. Personalization is the goal of that read.

If gaze works as a prompt, help can arrive with less friction. A glance could tell the model what text matters. Privacy and accuracy are clear concerns to test. The study is an early probe of that channel. It may shape how glasses assist in daily reading. Hands-free help is the draw.

Catalog record

CREATORJiexin Ding, Yizhuo Zhang, Xinyun Liu, Ke Chen, Yuntao Wang, Shwetak Patel, Akshay Gadre
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17676

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GazeSummary: Exploring Gaze as an Implicit Prompt for Personalization in Text-based LLM Tasks Conference: The 27th International Workshop on Mobile Computing Systems and Applications; February 25–26, 2026; Atlanta, GA, USA The 27th International Workshop on Mobile Computing Systems and Applications (HotMobile ’26), February 25–26, 2026, Atlanta, GA, USA DOI: 10.1145/3789514.3792037 ISBN: 979-8-4007-2471-8/2026/02 Jiexin Ding, Yizhuo Zhang, Xinyun Liu, Ke Chen, Yuntao Wang, Shwetak Patel, Akshay Gadre Abstract . Smart glasses are accelerating progress toward more seamless and personalized LLM-based assistance by integrating multimodal inputs. Yet, these inputs rely on obtrusive explicit prompts. The advent of gaze tracking on smart devices offers a unique opportunity to extract implicit user intent for personalization. This paper investigates whether LLMs can interpret user gaze for text-based tasks. We evaluate different gaze representations for personalization and validate their effectiveness in realistic reading tasks. Results show that LLMs can leverage gaze to generate high-quality personalized summaries and support users in downstream tasks, highlighting the feasibility and value of gaze-driven personalization for future mobile and wearable LLM applications. † † cc-license: by 1. Introduction Smart glasses with see-through displays and AI assistants are bringing in a new era of mobile computing. Beyond traditional metrics like throughput, latency, and battery life, these devices increasingly focus on delivering personalized user experiences. Current designs rely on explicit prompts via microphones paired with LLMs such as ChatGPT, Gemini, and Grok. While effective, these prompts require user effort and are often cumbersome or impractical. For example, a user attending a research talk would need to repeatedly say “this slide is important” to guide a summarization system, which is intrusive and unnatural. Gaze tracking is rapidly becoming standard in XR headsets and is widely expected to be included in the next generation of smart glasses (Project Aria), making it a practical implicit input for LLM-based tasks. This paper focuses specifically on the above task of generating a personalized summary based on user gaze. Prior work has shown that gaze can effectively reflect user attention ( 6 ) and can support extractive summarization through gaze density ( 21 ) , classifiers ( 19 ) , or pre-trained models ( 24 ; 8 ) . However, no existing work integrates gaze directly with LLMs leveraging their strengths in text-based tasks. Figure 1 . By integrating gaze as a proxy for user attention, our approach enables large language models to produce summaries centered on the content users find most relevant to their interest. This paper, GazeSummary, examines how LLMs can interpret gaze data to generate summaries that align with users’ attention. While prior work shows it is feasible to personalize summary using gaze, most approaches use gaze only to locate relevant text and have not explored how to integrate gaze directly into LLMs and what is the best representation of gaze for such integration . We address this gap by evaluating multiple gaze representations, finding that gaze heatmap performs best in its similarity with user-provided summaries. It captures fine-grained attention patterns thus outperform explicit focus inputs, particularly at the phrase level. We further explore the potential of gaze-based personalization in enhancing real-world user experience and productivity by developing a prototype gaze-guided assistant for academic reading. It automatically generates personalized summaries for users based on gaze heatmaps. We conducted a user study with 10 participants that compares the effectiveness of GazeSummary with traditional methods , such as summarization based on explicit user prompts and text-only input, in creation-oriented tasks like writing. The results show that GazeSummary outperforms text-only summarization and achieves comparable performance to user-prompt-based summarization, while requiring lower mental effort to use. The main takeaways of this work are as follows: • We explore several gaze representations as the input for LLM for personalized text summarization and find heatmap outperforms others and even the explicit-prompt-based summary at phrase level. • We build a gaze-guided assistant for academic reading and conduct a user study which shows that GazeSummary is more helpful and easy-to-use compared to the commonly used summarization methods. 2. Related Work Gaze Tracking and LLMs: With the rapid development of large language models (LLMs), recent research has begun exploring ways to integrate gaze signals into LLMs. Some works like GazeNoter ( 20 ) and EmBARDiment ( 4 ) treats gaze purely for selecting outputs or objects of importance, and incorporates this information into the LLM prompt, resulting in more efficient and context-aware interactions. Other works have treated gaze as a visual modality and input it into vision-language models (VLMs) to utilize the models’ visual encoder to interpret gaze. By aligning gaze with the real world, these approaches ( 22 ; 23 ; 14 ) achieve more efficient visual question answering (VQA) solutions that incorporate image features with queries to respond to the user. While these approaches have combined gaze and LLMs for interaction and VQA tasks, text-based scenarios are very different: gaze data is much denser, and text-related tasks require finer granularity, often at the word level. To the best of our knowledge, no work has systematically investigated optimum gaze representations for LLMs to understand and leverage in text-related tasks, such as summarization. Personalized Text Summarization: Summaries condense information to highlight key ideas, enhancing understanding and retention ( 5 ) . However, generic summaries often fail to match what individual users consider important, highlighting the need for personalized summaries ( 2 ) . Previous works on personalization determine user context based on the background information ( 13 ; 1 ; 25 ) or users’ explicit feedback ( 10 ) . However, these approaches mainly rely on past information about the user and thus cannot dynamically adapt summaries to reflect the user’s current reading state and focus. Given the strong links between gaze patterns and cognitive states ( 18 ; 16 ; 12 ) , some works use gaze to detect content of interest during reading and compile it into personalized summaries. Research shows that re-reading important parts and spending more time on them improves comprehension, and are reflected in gaze data ( 18 ; 11 ) . Based on this, some methods generate personalized summaries by selecting the most focused sentences or UIs according to gaze density ( 21 ; 8 ; 24 ) , while others aggregate gaze into features such as fixations to identify key sentences ( 19 ) . However, most of these approaches simply concatenate key sentences or use basic word embeddings, resulting in summaries that are less fluent and coherent than generic ones. 3. Gaze Representations This section explores the best representation of gaze information for text-based LLM tasks using traditional encoding methods. Currently, there is no consensus on the optimal way to encode gaze features via LLM prompts for summarization tasks. Existing approaches vary significantly in terms of the type and amount of gaze information preserved, as well as how explicitly that information is encoded within the model input. We identify two key dimensions along which gaze representations differ: (1) Information richness : the extent to which raw or derived gaze features (e.g., fixation duration, saccade pattern, regressions) are preserved. (2) Visual explicitness : the degree to which gaze signals are encoded in a way that is easily interpretable by models. Based on these criteria, we analyzed existing gaze

Read the complete text