Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention DOI: TBD Conference: Proceedings of the 27th ACM International Conference on Multimodal Interaction; October 13–17, 2025; Canberra, Australia Proceedings of the 27th ACM International Conference

Omar Elgohary, Zhu-Tian Chen · 2025

Read Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention DOI: TBD Conference: Proceedings of the 27th ACM International Conference on Multimodal Interaction; October 13–17, 2025; Canberra, Australia Proceedings of the 27th ACM International Conference by Omar Elgohary, Zhu-Tian Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05898.txt.

Introduction

Reading on screens that respond to eyes and hands rather than a trackpad is tested in a 2025 study by Omar Elgohary and one co-author. Multimodal interfaces are spreading in education and assistive technology, yet their effect on memory is understudied. The authors ask whether gaze plus gesture control helps or hurts retention.

Twelve participants read four articles through both a standard trackpad setup and an interface using eye movement and hand gestures to scroll, zoom, and reveal content. Retention was tested at once and again after a day, alongside workload and usability scales. Targeted retention matched the trackpad at first, but the multimodal version demanded more mental effort and seemed to fade faster over the day. The authors draw design lessons for learning tools that use richer input.

Catalog record

CREATOROmar Elgohary, Zhu-Tian Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-05898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention, Action, and Memory: How Multi-modal Interfaces and Cognitive Load Alter Information Retention DOI: TBD Conference: Proceedings of the 27th ACM International Conference on Multimodal Interaction; October 13–17, 2025; Canberra, Australia Proceedings of the 27th ACM International Conference on Multimodal Interaction (ICMI ’25), October 13–17, 2025, Canberra, Australia ISBN: TBD CCS: Human-centered computing User studies Omar Elgohary, Zhu-Tian Chen Abstract . Each year, multi-modal interaction continues to grow within both industry and academia. However, researchers have yet to fully explore the impact of multi-modal systems on learning and memory retention. This research investigates how combining gaze-based controls with gesture navigation affects information retention when compared to standard track-pad usage. A total of twelve participants read four textual articles through two different user interfaces which included a track-pad and a multi-modal interface that tracked eye movements and hand gestures for scrolling, zooming, and revealing content. Participants underwent two assessment sessions that measured their information retention immediately and after a twenty-four hour period along with the NASA-TLX workload evaluation and the System Usability Scale assessment. The initial analysis indicates that multi-modal interaction produces similar targeted information retention to traditional track-pad usage, but this neutral effect comes with higher cognitive workload demands and seems to deteriorate with long-term retention. The research results provide new knowledge about how multi-modal systems affect cognitive engagement while providing design recommendations for future educational and assistive technologies that require effective memory performance. 1. Introduction The mixture of eye-tracking and free-space gesturing as a means of interactivity has been researched rather thoroughly ( Yuan et al. 2025 ) . Each modality has its advantages as well as disadvantages. For instance, the eye gaze easily and effortlessly reflects user focus and aids in the fast target acquisition, whereas free space gestures provide a variety dynamic change possibilities. In contrast, free-space gestures offer no feedback support and eye tracking suffers from the natural jitter and inaccuracy of fine control movements ( Mirza and Basu 2015 ) . Recent studies emphasize that integrating gaze and gesture systems produce smoother and finer interactions. This is due to the leveraging the strengths of each ( Mirza and Basu 2015 ; Wibirama et al. 2020 ; Yuan et al. 2025 ) . These results range from surpassing to matching the performance of the traditional techniques. However, these multi-modal systems introduce a higher cognitive load, as users must simultaneously manage multiple input channels, coordinate visual attention, and perform precise motor actions, often without explicit feedback mechanisms ( Yuan et al. 2025 ) . Cognitive Load Theory describes the constraints of working memory on a particular task and the stress it incurs while emphasizing the limited cognitive resources available ( Yuan et al. 2025 ; Kosch et al. 2023 ; Kopp and Bergmann 2017 ) . It especially concerns itself with three types of load: intrinsic, which accompanies the complexity of the material to be learned, extraneous which is a result of the design and presentation of the materials as well as the structure of the activities, and germane load which is the effort exerted in recalling or learning the information ( Kopp and Bergmann 2017 ; Sweller et al. 2011 ) . In turn, multi-modal interaction systems increase cognitive challenges with their complexity and require cross-modal coordination ( Yuan et al. 2025 ; Kosch et al. 2023 ) . Given the interest from the industry and academia, more research is needed on the impact of multi-modal interaction on cognitive processes such as learning and retention of information. This paper seeks to assess whether the retention of users’ information is better with a multi-modal gaze and gesture interface compared to a standard track-pad interface. In order to assess this, participants were asked to engage with articles where important parts were textually shrunk, requiring greater usage of each system. Information recall metrics, cognitive load collected through the NASA-TLX, usability perception scores captured through the System Usability Scale, as well as behavioral metrics were collected as a baseline to analyze the effect multi-modal interactions have on cognitive performance. The main contributions of this paper are: • Design and implement a multi-modal reading interface that integrates real-time gaze tracking and gesture-based control for article information acquisition. • Conduct a within-subjects user study comparing gaze and gesture interaction against traditional interfaces in terms of immediate and delayed information retention, cognitive load, and usability perceptions. • Present empirical evidence showing that multi-modal interaction is on-par with traditional interaction methods, even with increased mental demands. 2. Related Work Related work to our gaze+gesture system and user study can be divided into two primary categories: multi-modal gaze+gesture input systems and cognitive load theory with its relation to information retention. 2.1. Multi-modal Gaze and Gesture Systems There has been growing interest in leveraging multi-modal interactions that combine gaze and gesture input due to their potential to create more natural, expressive, and hands-free experiences. Recent advances in machine learning have enabled standard webcams to approximate the accuracy traditionally reserved for infrared (IR) eye-tracking systems, making gaze interactions increasingly accessible without expensive specialized hardware ( Papoutsaki et al. 2016 ; Apple Inc. 2024 ; Sightful Ltd. 2024 ; Yuan et al. 2025 ) . At the same time, improvements in computer vision have made gesture and gaze recognition more intuitive and deployable with low-cost sensors, with these interfaces now appearing in domains such as the automotive industry, AR/VR headsets, and even general phone usage ( L R D et al. 2022 ; Chen et al. 2022 ; Pfeuffer et al. 2017 ; Apple Inc. 2024 ) . Prior systems have explored gaze-supported or gesture-supported input individually, demonstrating benefits for target acquisition and remote control tasks ( Mirza and Basu 2015 ) . Combined gaze and gesture interaction promises even greater expressiveness by using gaze to indicate attention and gestures to confirm or manipulate targets ( Mirza and Basu 2015 ) . However, most multi-modal implementations still run into issues, including gaze-tracking inaccuracies, the need for calibration, and gesture mis-recognition when deployed in unconstrained environments ( Yuan et al. 2025 ) . Although some systems have integrated gaze and gesture successfully for tasks such as large display interaction or remote collaboration, few works have considered how the cognitive implications of multi-modal input impacts learning and retention in users. Most prior studies focus on interaction performance without examining effects on user cognitive load, learning, and/or retention. In addition, many mainly rely on NASA-TLX assessments to draw their conclusions ( Kosch et al. 2023 ) . 2.2. Cognitive Load and Learning Retention in Multi-modal Systems Cognitive load considerations have become an integral component of planning any new multi-modal system. There have been numerous evaluation approaches, such as the NASA-TLX, that have been widely applied in the discipline. The NASA-TLX has tirelessly been recognized as the most important and widely accepted measure of user cognitive load in the industry. The more recent studies, such as the one conducted by Kosch et al. ( Kosch et al. 2023 ) , were able to highlight quite effectively how NASA-TLX becomes an outdated model of HCI in multi-modal evaluation. In which Kosch et al. ( Kosch et al. 2023 ) , argued that HCI multi-modal studies require more innovative and sophisticated evaluation methods that

Read the complete text