Distinguishing Target and Non-Target Fixations with EEG and Eye Tracking in Realistic Visual Scenes Conference: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION; November 4–8, 2024; San Jose, Costa Rica INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI ’24), November 4–8, 2024, San Jose, Co

Mansi Sharma, Camilo Andrés Martínez Martínez, Benedikt Emanuel Wirth, Antonio Krüger, Philipp Müller · 2025

Read Distinguishing Target and Non-Target Fixations with EEG and Eye Tracking in Realistic Visual Scenes Conference: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION; November 4–8, 2024; San Jose, Costa Rica INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI ’24), November 4–8, 2024, San Jose, Co by Mansi Sharma, Camilo Andrés Martínez Martínez, Benedikt Emanuel Wirth, Antonio Krüger, Philipp Müller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01853.txt.

Introduction

Mansi Sharma and co-authors test whether brain waves plus eye tracking can tell a searcher's target from passing glances, in a 2025 paper. Knowing what a user is actually looking for is a base skill for assistance systems. Earlier studies used staged searches with abstract images and no scene context. Real search is guided by the scene itself.

Thirty-six participants searched freely through one hundred forty realistic scenes. Eye movements and scalp signals were recorded together. Classifiers learned to separate fixations on the target from all others. Success here would let interfaces infer intent during natural looking, a step toward help that arrives before it is requested.

Catalog record

CREATORMansi Sharma, Camilo Andrés Martínez Martínez, Benedikt Emanuel Wirth, Antonio Krüger, Philipp Müller
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01853

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distinguishing Target and Non-Target Fixations with EEG and Eye Tracking in Realistic Visual Scenes Conference: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION; November 4–8, 2024; San Jose, Costa Rica INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI ’24), November 4–8, 2024, San Jose, Costa Rica DOI: 10.1145/3678957.3685728 ISBN: 979-8-4007-0462-8/24/11 CCS: Human-centered computing CCS: Human-centered computing User studies Mansi Sharma, Camilo Andrés Martínez Martínez, Benedikt Emanuel Wirth, Antonio Krüger, Philipp Müller Abstract . Distinguishing target from non-target fixations during visual search is a fundamental building block to understand users’ intended actions and to build effective assistance systems. While prior research indicated the feasibility of classifying target vs. non-target fixations based on eye tracking and electroencephalography (EEG) data, these studies were conducted with explicitly instructed search trajectories, abstract visual stimuli, and disregarded any scene context. This is in stark contrast with the fact that human visual search is largely driven by scene characteristics and raises questions regarding generalizability to more realistic scenarios. To close this gap, we, for the first time, investigate the classification of target vs. non-target fixations during free visual search in realistic scenes. In particular, we conducted a 36-participants user study using a large variety of 140 realistic visual search scenes in two highly relevant application scenarios: searching for icons on desktop backgrounds and finding tools in a cluttered workshop. Our approach based on gaze and EEG features outperforms the previous state-of-the-art approach based on a combination of fixation duration and saccade-related potentials. We perform extensive evaluations to assess the generalizability of our approach across scene types. Our approach significantly advances the ability to distinguish between target and non-target fixations in realistic scenarios, achieving 83.6% accuracy in cross-user evaluations. This substantially outperforms previous methods based on saccade-related potentials, which reached only 56.9% accuracy. Figure 1 . (a) Previous work: Simplistic search displays with instructed search trajectory (b) Our work: Realistic search scenes with free search trajectory, search target: Wrench, Outlook (a) Previous work: Simplistic search displays (b) Our work: Realistic search scenes. 1. Introduction Visual search is ubiquitous in daily life. It occurs both when we search for items such as tools in a cluttered physical environment, but also in human-computer interaction when e.g. searching for a specific application icon. Due to its ubiquity, assistance during visual search can potentially have a large impact on user satisfaction and task completion times ( Boot et al. 2009 ; Guo et al. 2022 ) . For instance, an assistive system that knows that the user is currently fixating on the icon of an application they want to open can automatically launch this application for the user, overcoming the Midas touch problem in gaze-based interaction ( Elmadjian and Morimoto 2021 ; Mohan et al. 2018 ) . On the other hand, a system that is supposed to help the user in locating a tool in a cluttered environment only needs to be activated when the user has not yet found the desired object. The foundation for such novel assistive systems is the ability to distinguish fixations on the search target ( target fixations ) from fixations that are not on the search target ( non-target fixations ). Previous work documented the feasibility to classify target from non-target fixations based on electroencephalography (EEG) ( Brouwer et al. 2013 ) , as well as a combination of EEG- and eye tracking features ( Brouwer et al. 2017 ) , with a combination of saccade-related EEG potentials and a fixation duration feature being most promising. However, these previous studies are limited in two key aspects: First, they utilized simplistic abstract stimuli such as Landolt Cs and strings of symbols hidden by (’#’) on uniform backgrounds, disregarding any scene context. Second, they did not let users search freely for the target, but explicitly instructed search trajectories. This is in stark contrast to application scenarios where user search freely in a complex, structured scene such as in a workshop, or on a computer desktop. Indeed, research in psychology indicates that scene context has an important influence on visual search ( Castelhano and Henderson 2007 ; Hollingworth 2009 ; Everett and Byrne 2004 ; Jin et al. 2023 ) . It therefore remains an open question to what extent the results achieved with simple stimuli on uniform backgrounds can be transferred to more realistic stimuli placed in complex, cluttered scenes. In our work we for the first time study the classification of target vs. non-target fixations from EEG and eye tracking recordings during free visual search in complex scenes. In particular, we study two highly application-relevant scenarios: searching for a tool in a cluttered workshop, and searching for an icon on a desktop screen. In a 36-participant user study we recorded gaze behavior and EEG measurements of people performing a visual search task in 140 different, diverse scenes from these two scenarios. We present an automatic approach to distinguish target from non-target fixations based on state-of-the-art Common Spatial Pattern (CSP) EEG features and a fixation duration feature. Our approach reaches 83.6% accuracy in cross-user evaluation and clearly outperforms the approach based on saccade-related potentials used in previous work for target vs. non-target fixation classification (56.9% accuracy) ( Brouwer et al. 2017 ) . Furthermore, we perform extensive evaluations on the impact of scene type on classification performance. Our results indicate that to reach maximum performance it is crucial to train on data from the same scene domain, and adding data of additional scenes during training did not lead to improvements. 2. RELATED WORK The present work is related to studies investigating scene influences on visual search as well as to previous approaches to classifying target vs. non-target fixations. 2.1. Visual Search and Scene Context Traditionally, cognitive research on visual search focused on highly-controlled, but artificial target and non-target stimuli that were presented in specific spatial configurations on blank backgrounds ( Folk et al. 1992 ; Treisman and Gelade 1980 ; Theeuwes 1992 ) . However, starting in the late 2000s, an increasing number of studies investigated search for real objects in naturalistic scenes ( Castelhano and Henderson 2007 ; Hollingworth 2009 ; Torralba et al. 2006 ) . This research argues that when viewing a scene, observers can quickly derive the "gist" of that scene via a non-selective pathway, that is, a pathway that does not rely on identification of individual objects ( Wolfe et al. 2011 ) . The information extracted from this scene gist, in turn, guides attention based on syntactic and semantic principles ( Wolfe 2020 ; Wolfe and Horowitz 2017 ) . Syntactic guidance refers to physical constraints that describe structurally plausible locations for specific objects. For example, if we assume that observers are searching for a hammer in the left scene depicted in Figure 1 (b), the observers will likely not search for the hammer in front of the gray brick wall or in front of the red cupboard because the hammer would fall to the ground due to gravity. Instead, observers would more likely search on horizontal surfaces or on vertical surfaces where tools can be attached (like a pegboard). Semantic guidance refers to the meaningfulness of an object in a specific location. For example, observers would be more likely to search for the hammer on

Read the complete text