A deep learning approach to track eye movements based on events
2025
Read A deep learning approach to track eye movements based on events (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04827.txt.
Introduction
Chirag Seth, Divya Naiken, and Keyan Lin wrote a computing paper in 2025 on tracking eye movement with a special camera that records change rather than full frames. Human eyes move very fast, and standard tracking needs costly high-speed cameras. The team aimed to find the center of the eye from the camera's event stream. Such tracking matters for virtual and augmented reality gear.
The authors tried several learning methods, and a combined model using both image layers and memory layers worked best. It reached about eighty-one percent accuracy in locating the eye center. The team proposes follow-up work on a method that shows which inputs drive each output, to make the model easier to trust and read.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04827
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A deep learning approach to track eye movements based on events Abstract This research project addresses the challenge of accurately tracking eye movements during specific events by leveraging previous research. Given the rapid movements of human eyes, which can reach speeds of 300°/s, precise eye tracking typically requires expensive and high-speed cameras. Our primary objective is to locate the eye center position (x, y) using inputs from an event camera. Eye movement analysis has extensive applications in consumer electronics, especially in VR and AR product development. Therefore, our ultimate goal is to develop an interpretable and cost-effective algorithm using deep learning methods to predict human attention, thereby improving device comfort and enhancing overall user experience. // To achieve this goal, we explored various approaches, with the CNN_LSTM model proving most effective, achieving approximately 81% accuracy. Additionally, we propose future work focusing on Layer-wise Relevance Propagation (LRP) to further enhance the model’s interpretability and predictive performance. 1 Introduction Eye-tracking methods have broad applications in domains including consumer electronics, neuroscience and psychology, as it has the potential to understand human visual behavior [1]. Traditional eye-tracking systems typically require specialized hardware to capture eye movements, and analyze massive data in the process. In comparison, event-based eye-tracking only capture sparse data when eye moves, which significantly reduces processing time and power consumption. In consumer electronics, particularly AR/VR, using event-based eye-tracking methods enable lighter, smaller and more comfortable devices. In neuroscience, such systems aid in deciphering visual attention processes and diagnosing neurological disorders, enhancing our understanding of cognitive mechanisms. Our project aims to deploy some popular neural network structures in event-based eye-tracking, compare effectiveness and explore the potential of model interpretability. 2 Literature Review 2.1 Models Convolutional Neural Networks (CNNs) have been widely used in event-based eye-tracking research due to their ability to extract features from raw input data. Recurrent Neural Networks (RNNs) are another common architectures in event-based eye-tracking methods. Compared with CNNs, RNNs have the ability to capture temporal dependencies in sequential data by maintaining an internal state or memory. This makes them well-suited for processing event streams. Within event-based eye-tracking, architectures such as Gated Recurrent Units (GRUs), Long Short-Term Memory (LSTM) networks, and Bidirectional LSTMs (Bi-LSTMs) are frequently utilized and demonstrate commendable performance. In our project, we will harness the synergistic potential of CNNs and RNNs by integrating them into a cohesive framework. By combining the strengths of both architectures, we aspire to achieve heightened levels of accuracy [2]. Moreover, one distinguishing aspect of our research is the utilization of data captured from the DVXplorer Mini event camera. This choice sets us apart from many previous studies that rely on different eye-tracking devices or datasets. The DVXplorer Mini event camera offers several advantages, including high temporal resolution, low power consumption, and compact size, making it ideal for capturing fine-grained eye movement data in real-time. By leveraging data from the DVXplorer Mini event camera, our research benefits from the unique qualities of this device, enabling us to analyze eye movement patterns with unprecedented precision and efficiency. This distinct dataset allows us to explore new avenues in event-based eye-tracking research and provides insights that were previously inaccessible with traditional eye-tracking methods. In summary, our research stands out by combining CNNs and RNNs in a novel framework while utilizing data from the DVXplorer Mini event camera. This integrated approach, coupled with the unique qualities of the dataset, positions our research at the forefront of interpretable eye-tracking models and opens new possibilities for understanding and analyzing eye movement data. 2.2 Interpretation One main challenge of eye-tracking technique is interpretability. The unclear nature of neural networks poses a significant obstacle to the interpretation and comprehension of models trained on eye movement datasets, hence restricting their usefulness in domains that necessitate explainability [3]. This project aims to develop an optimized, transparent, and interpretable model for predicting pupil coordinates, employing explainable AI techniques to ensure the prediction process is fully understandable. A specific challenge with models trained on eye movement data, as identified in, Challenges in Interpretability of Neural Networks for Eye Movement Data concerns the interpretability of data points. The research has highlighted the difficulties surrounding understanding the relationship between data points and the representations of feature vectors created by the model within its hidden layers. The focus of interpretability tools should, therefore, shift toward dissecting the behavior of these hidden states. By doing so, it becomes possible to examine how these vectors encapsulate the attributes of the input, thereby enhancing the ability to make the model’s learning process understandable. 3 Methodology In recent studies, the amalgamation of Convolutional Neural Networks (CNNs) with recurrent models like Gated Recurrent Units (GRU) or Long Short-Term Memory (LSTM) networks has shown promising results. This fusion offers a unique advantage in feature extraction and temporal modeling, particularly beneficial for tasks involving sequential data analysis. Specifically, in the realm of eye movement detection, CNNs exhibit proficiency in extracting spatial features from event frames, shedding light on intricate eye movement patterns. Concurrently, recurrent models excel in capturing temporal dependencies within sequential data, enabling dynamic learning and adaptation over time. 3.1 Data Processing and Preparation Effective model preparation commences with rigorous data processing to ensure compatibility with the chosen architecture. For this research, our dataset[4] comprises recordings from 13 subjects, spanning multiple recording sessions per subject. These sessions encompass a diverse range of activities, including random movements, saccades, text reading, smooth pursuit, and blinks. Each session generates raw event data, subsequently transformed into event frames for comprehensive analysis. With a dataset volume of approximately 1 GB in compressed .h5 format, diverse challenges emerge in data representation and processing pipelines, contingent upon the chosen methodology and analytical approach. Figure 1: Figure Showing the flow of data processing steps 3.2 Labelling Scheme The dataset’s ground truth labeling operates at a frequency of 100Hz, encompassing two primary components for each label (x, y, close): pupil center coordinates (x, y) and a binary ’close’ indicator denoting eye blinks (0 for eye opening, 1 for eye closing). For the training split, labels are initially provided at 100Hz, offering the flexibility to downsample as per specific requirements. In our provided training sample pipeline, labels undergo downsampling to 20Hz, with the exclusion of the ’close’ label by default. Evaluation of the test set is uniformly conducted at a 20Hz frequency, ensuring consistency in evaluation metrics and results. Any modifications to the data loading pipeline should carefully consider this downsampling requirement to maintain evaluation integrity and coherence. 3.3 Different Approaches we Tried On Our Problem Figure 2: Figure Showing the high level architecture of our model 3.3.1 Model Summary: CNN-GRU for Eye Tracking Utilizing a CNN followed by a single-layer GRU, this model architecture integrates spatial and temporal information, making it essential for sequential eye tracking tasks. It is particularly suitable for eye tracking scenarios where understanding both spatial patterns and temporal changes over time is crucial. 3.3.2 Model Summary: CNN-BiLSTM for Eye Tracking The architecture of this model involves employing a CNN with a single-layer bidirectional LSTM (BiLSTM). It offers advantages by combining convolutional and bidirectional recurrent layers, effectively capturing both spatial and temporal features. This makes it well-suited for eye tracking tasks requiring comprehensive analysis of both spatial and temporal dynamics.