Neural network-based encoding in free-viewing fMRI with gaze-aware models
Dora Gözükara, Nasir Ahmad, Katja Seeliger, Djamari Oetringer, Linda Geerligs · 2026
Read Neural network-based encoding in free-viewing fMRI with gaze-aware models by Dora Gözükara, Nasir Ahmad, Katja Seeliger, Djamari Oetringer, Linda Geerligs (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11663.txt.
Introduction
Gozukara and co-authors adapt brain encoding models for free-viewing fMRI. Data sets used central fixation, which strays from natural vision and loads the task. Their gaze-aware models fold eye-tracking data into the feature streams. For each fixation, the model samples only the local, timely parts of the CNN feature map. The data come from StudyForrest, a task-free natural movie viewing set. Gaze-aware models match standard ones with 112 times fewer parameters. Gains run largest for viewers whose eyes move most.
Catalog record
| CREATOR | Dora Gözükara, Nasir Ahmad, Katja Seeliger, Djamari Oetringer, Linda Geerligs |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54703 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural network-based encoding in free-viewing fMRI with gaze-aware models Dora Gözükara, Nasir Ahmad, Katja Seeliger, Djamari Oetringer, Linda Geerligs Abstract Representations learned by convolutional neural networks (CNNs) exhibit a remarkable resemblance to information processing patterns observed in the primate visual system on large neuroimaging datasets collected under diverse, naturalistic visual stimulation, but with instruction for participants to maintain central fixation. This viewing condition, however, diverges significantly from ecologically valid visual behaviour, suppresses activity in visually active regions, and imposes substantial cognitive load on the viewing task. We present a modification of the encoding model framework, adapting it for use with naturalistic vision datasets acquired under fully natural viewing conditions, without fixation, by incorporating eye-tracking data. Our gaze-aware encoding models were trained on the StudyForrest dataset, which features task-free naturalistic movie viewing. By combining eye-tracking data with the visual content of movie frames, we generate combined subject-wise gaze-stimulus specific feature time series. These time series are constructed by sampling only the locally and temporally relevant elements of the CNN feature map for each fixation. Our results demonstrate that gaze-aware encoding models match the performance of conventional encoding models with 112 × \times fewer model parameters. Gaze-aware encoding models were especially beneficial for participants with more dynamic eye-movement patterns. Therefore, this approach opens the door to more ecologically valid models that can be built in more naturalistic settings, such as playing games or navigating virtual environments. \papertype Original Article \paperfield Journal Section \corraddress Dora Gözükara \corremail [email protected] keywords fMRI, CNN , encoding, naturalistic, eye-tracking 1 Introduction Cognitive neuroscience is increasingly adopting integrative research paradigms that combine naturalistic experimental designs with advanced analytical tools. For instance, the use of complex, multimodal and naturalistic stimuli in experiments, paired with the application of artificial neural networks (ANNs), has gained traction in recent years. Representations learned by convolutional neural networks (CNNs), in particular, have become a prominent model for understanding stages of information processing in the primate visual system [ 14 , 33 ] . These developments align with a broader effort to make experimental designs and analytical methods more ecologically valid. This paper builds on these advances by revisiting assumptions underlying brain-encoding models that rely on CNN features. Specifically, we propose that incorporating natural eye movements into the modelling process can significantly enhance the ecological validity of the underlying data and reduce the parameter spaces of models. By integrating gaze dynamics we demonstrate that CNN-based brain encoding can accommodate naturalistic stimuli without the need for restrictive fixation in the experimental designs. 1.1 Naturalistic Experimental Design Natural visual behaviour is inherently and consistently active, with frequent gaze shifts to explore salient features in the environment [ 21 ] . Despite this, almost all experiments underlying recent deep neural network (DNN)-based brain modelling studies have imposed central fixation constraints during neuroimaging experiments to control attentional focus and minimize variability introduced by eye movements [ 27 , 36 , 24 , 4 , 8 , 22 ] . While this approach ensures stimulus-response measurements that are simple to handle analytically, it diverges from natural viewing behaviour and imposes substantial cognitive load. Fixation protocols also suppress activity in visually dynamic brain regions, potentially skewing insights into how the brain processes naturalistic stimuli. For example, 15 highlight that fixating on a static location during movie viewing disrupts the natural flow of perception. Nevertheless, some large neuroimaging data collections required participants to maintain fixation for extended periods, up to several dozens of hours [ 38 , 4 ] . Even in studies incorporating eye-tracking data, its use is typically limited to verifying fixation compliance or assessing participant alertness [ 26 , 49 ] . Here we propose to use eye-tracking data to inform feature selection in CNN-based brain-encoding models. This approach allows for fixation-free experiments that better capture the dynamic and active nature of visual behaviour, offering a path toward more ecologically valid experimental designs and lowered computational burdens. This research is particularly relevant since some of the largest ongoing neuroimaging data recording efforts have decided against fixation protocols for aforementioned reasons [ 7 ] . 1.2 Holistic Analysis Advances in artificial neural networks, particularly CNNs, have revolutionized our ability to model brain activity. Early brain-encoding models relied on manually engineered features or simple statistical representations [ 27 , 36 ] . By contrast, CNNs learn hierarchical visual representations directly from large-scale naturalistic data, making them a powerful tool for studying information processing in the visual system [ 30 , 33 , 14 ] . One key similarity between CNNs and brains lies in their use of spatially selective receptive fields, where individual neurons or units respond to specific regions of the visual field. Most fMRI studies using CNN-based encoding models do not leverage these spatial properties and pool features from entire CNN layers into single feature vectors to predict voxel activity [ 2 , 1 , 18 , 24 , 48 , 19 , 25 ] . Limited exceptions exist, such as recent works which transform input images to mimic cortical magnification of the visual input [ 10 ] . The use of features gathered from across all spatial locations in CNN layers inflates model parameter spaces, requiring large datasets to achieve reliable fits and introducing ambiguities in feature selection, given that there are likely multiple combinations of features and locations that could explain the signal of a voxel. By sampling CNN features with eye-movements instead, we propose a significant reduction in parameter space complexity, improving both data efficiency and model interpretability. 1.3 Gaze-Aware Brain Encoding The primary goal of this work is to demonstrate the efficacy of combining the power of convolutional neural networks (CNNs) and eye-movement data to model the brain’s visual processing system in experimental conditions that are closer to natural viewing scenarios. Specifically, in experimental neuroimaging scenarios that allow for eye movements. We further contribute to modelling efficiency by introducing a method that modifies brain-encoding models to significantly reduce the parameter space and thus the amount of data required for training. To demonstrate this, we used the publicly available StudyForrest dataset [ 20 ] , which includes approximately two hours of fixation-free movie-watching runs and comprehensive eye-tracking data during the movies. Using a CNN pre-trained for natural image classification, we extracted features from all convolutional layers in response to the movie frames. This generated a time series of features for each spatial location and layer corresponding to each movie frame. We sampled these features using subject-specific gaze data to construct personalized feature time series. The resulting feature time series were highly efficient in size, reducing feature data size by factors ranging from 112-fold for the smallest layer to approximately 30,000-fold for the largest layer. Finally, we trained linear encoding models to learn optimal feature combinations from these reduced temporal feature series that predict voxel activity. By integrating gaze dynamics, this pipeline not only addresses the variability across and within individuals but could also enable robust predictions using substantially smaller datasets. This represents a meaningful step forward in making neural encoding models more efficient and practical for real-world applications, particularly in scenarios where traditional experimental constraints, such as fixation, are relaxed. 2 Methods 2.1 fMRI Dataset We make use of the open ‘StudyForrest’ dataset which, alongside the