SIM: S urface-based fMRI Analysis for I nter-Subject M ultimodal Decoding from Movie-Watching Experiments
Simon Dahan, Gabriel Bénédict, Logan Z. J. Williams, Yourong Guo, Daniel Rueckert, Robert Leech, Emma C. Robinson · 2025
Read SIM: S urface-based fMRI Analysis for I nter-Subject M ultimodal Decoding from Movie-Watching Experiments by Simon Dahan, Gabriel Bénédict, Logan Z. J. Williams, Yourong Guo, Daniel Rueckert, Robert Leech, Emma C. Robinson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16471.txt.
Introduction
Simon Dahan and co-authors present, in twenty twenty-five, a method to decode what a person is watching or hearing from their brain scans, even across people and unseen movies. Brain decoders usually train and test on the same data, which limits use in brain-computer interfaces. Brains differ in cortical layout, so signals are hard to align across people. Their method, SIM, uses surface vision transformers. These encode the layout of cortical networks and their interplay as a moving image on the brain's surface.
They align three modes, audio, video, and fMRI, with contrastive learning, so each can retrieve the others. Tests used high-field scans from one hundred seventy-four healthy adults watching movies. The system could tell which clips a person watched, even for new people and new movies.
Catalog record
| CREATOR | Simon Dahan, Gabriel Bénédict, Logan Z. J. Williams, Yourong Guo, Daniel Rueckert, Robert Leech, Emma C. Robinson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16471
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SIM: S urface-based fMRI Analysis for I nter-Subject M ultimodal Decoding from Movie-Watching Experiments Simon Dahan, Gabriel Bénédict, Logan Z. J. Williams, Yourong Guo, Daniel Rueckert, Robert Leech, Emma C. Robinson Abstract Current AI frameworks for brain decoding and encoding, typically train and test models within the same datasets. This limits their utility for brain computer interfaces (BCI) or neurofeedback, for which it would be useful to pool experiences across individuals to better simulate stimuli not sampled during training. A key obstacle to model generalisation is the degree of variability of inter-subject cortical organisation, which makes it difficult to align or compare cortical signals across participants. In this paper we address this through use of surface vision transformers, which build a generalisable model of cortical functional dynamics, through encoding the topography of cortical networks and their interactions as a moving image across a surface. This is then combined with tri-modal self-supervised contrastive (CLIP) alignment of audio, video, and fMRI modalities to enable the retrieval of visual and auditory stimuli from patterns of cortical activity (and vice-versa). We validate our approach on 7T task-fMRI data from 174 healthy participants engaged in the movie-watching experiment from the Human Connectome Project (HCP). Results show that it is possible to detect which movie clips an individual is watching purely from their brain activity, even for individuals and movies not seen during training . Further analysis of attention maps reveals that our model captures individual patterns of brain activity that reflect semantic and visual systems. This opens the door to future personalised simulations of brain function. Code & pre-trained models will be made available at https://github.com/metrics-lab/sim , processed data for training will be available upon request at https://gin.g-node.org/Sdahan30/sim . 1 Introduction Figure 1: ( SIM ) seeks to align visual and audio stimuli - extracted from T=3 second movie clips - with brain activations acquired over the same time period: [A] 7T fMRI data, collected during movie watching, is first projected to subjects’ native cortical surfaces (resolution V =59292 vertices); then inflated to a sphere, and downsampled onto a regularly tessellated icosphere ( I_{6} with V =40962 vertices). [B] This is then encoded using a surface vision transformer (SiT), which tokenizes data by patching each I_{6} sphere with a lower-resolution icospheric grid to generate a sequence of triangular patches ( V =45 vertices per patch; N =1280 number of patches). [C] The SiT encoder ( \Phi^{fMRI}_{enc} ) is pre-trained as part of a video surface masked autoencoder (vsMAE) for fMRI-frame reconstruction; [D] and fMRI embeddings ( f_{i} ) are then aligned with CLIP contrastive training to video ( v_{i} ) [E] and audio ( a_{i} ) [F] embeddings learnt from a videoMAE ( \Phi^{\mathcal{V}}_{enc} ) and wav2vec ( \Phi^{\mathcal{A}}_{enc} ) model; and projected to vector spaces of common lenght by multimodal mappers ( f^{fMRI}_{\theta},f^{\mathcal{V}}_{\theta},f^{\mathcal{A}}_{\theta} ) . At test time this makes it possible to decode video/audio stimuli from fMRI (or vice versa) through comparing the similarity of their CLIP embeddings ( f_{i},a_{i},v_{i} ). This comparison generates a probability distribution over the candidate samples, allowing for evaluation of retrieval performance using top-K accuracy metrics. Over recent years, there has been growing interest in the extent to which machine learning frameworks, such as convolutional neural networks (CNNs) and transformers, can model neurological processes: from spatial encoding of the hippocampus ( Ellwood, 2024 ; Whittington et al., 2021 ; Kim et al., 2024 ) to replicating semantic ( Antonello & Huth, 2024 ; Caucheteux et al., 2023 ; Huth et al., 2016 ; Millet et al., 2022 ) and visual ( Benchetrit et al., 2023 ; Ozcelik & VanRullen, 2023 ; Tang et al., 2024 ; Wen et al., 2018 ) representations within the cortex. These approaches support the testing of new theories of human cognition ( Millet et al., 2022 ; Antonello & Huth, 2024 ; Caucheteux et al., 2023 ; Ellwood, 2024 ; Kriegeskorte, 2015 ; Whittington et al., 2021 ; Kim et al., 2024 ) , and allow for the encoding or decoding of stimuli ( Benchetrit et al., 2023 ; Défossez et al., 2023 ; Gu et al., 2022 ; Lindsay, 2021 ; Kriegeskorte, 2015 ; Ozcelik & VanRullen, 2023 ; Scotti et al., 2024 ; Thomas et al., 2022 ; Thual et al., 2023 ; Wen et al., 2018 ; Yamins & DiCarlo, 2016 ) from non-invasive human brain recordings such as functional magnetic resonance imaging (fMRI), magneto-encephalography (MEG) or electro-encephalography (EEG). However, such comparative models are subject-specific, typically require large amounts of imaging data per subject, and do not generalise to unseen individuals. This limits the extent to which they can be used to probe individual sources of variation, to ultimately build precision models of human cognition and behaviour. One contributing factor to the lack of generalisation of current models, is arguably that most models treat signals from adjacent locations in the brain as spatially independent - encoding patterns of brain activity using linear, or non-linear regression modules ( Huth et al., 2016 ; Millet et al., 2022 ; Caucheteux et al., 2021 ; Scotti et al., 2024 ; Ozcelik & VanRullen, 2023 ) . This ignores the well-documented spatial auto-correlation of signals across the cortex that is known to be behaviourally meaningful ( Bijsterbosch et al., 2018 ; Kong et al., 2019 ; Shinn et al., 2023 ; Leech et al., 2023 ; Margulies et al., 2016 ) . At a high level, cortical activity may be decomposed into signals from a relatively small number of functionally specialised areas ( Glasser et al., 2016a ; Smith et al., 2013 ) . These regions communicate through neuronal connections, resulting in coordinated patterns (or networks) of activity. Previous studies have shown that human brain function may be modelled from temporal sequences of these networks ( Vidaurre et al., 2017 ; Pervaiz et al., 2022 ; Smith et al., 2013 ) . This suggests that if we can build an encoding model that generalises these spatial patterns across individuals, then we can predict what any individual’s cortical functional dynamics should be in response to a given stimulus; or predict a novel stimulus from the spatial topography of their brain activity. With this in mind, recent efforts to generalise vision transformers (ViTs) ( Dosovitskiy et al., 2020 ) to the cortical surface present significant opportunity as they allow for the modelling long-range spatio-temporal interactions through the encoding of space-time self-attention ( Dahan et al., 2022 ; Dahan et al., 2024 ) . Relative to a 3D network of equivalent resolution, surface models are much more compact and require less memory; this creates scope for building more sophisticated network architectures. Moreover, surface modelling more faithfully represents the true geometry of the convoluted cortical surface, and for this reason has long been favoured for analysis of cortical signals ( Glasser et al., 2016b ; Margulies et al., 2016 ; Gordon et al., 2017b ; Gordon et al., 2017a ) . Contributions: In this paper, we introduce SIM (Fig 1 ), a novel framework that combines self-supervised contrastive learning with surface vision transformers (SiT) ( Dahan et al., 2022 ; Dahan et al., 2024 ) , to support multimodal decoding of audio+video (movie) stimuli from 7T cortical fMRI.