Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding

Yueh-Po Peng, Vincent K.M. Cheung, Li Su · 2025

Read Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding by Yueh-Po Peng, Vincent K.M. Cheung, Li Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22378.txt.

Introduction

Yueh-Po Peng, Vincent Cheung, and Li Su present a brain decoding model in a 2025 paper. Reading mental states from brain scans is a core goal of neuroscience, but scan data are noisy, huge, and scarce for any single person. The team borrows a trick from image AI: learn general patterns from a very large scan library first, then adapt to small tasks.

Their model, a transformer design, learned from scans of 995 people in a major public project. It splits attention between space and time, which keeps the computing cost in check. After this broad training, the model read task-driven brain activity far better than models trained from scratch, across tasks of sensing and thinking. Gains also came from wide spatial views and from training data that matched the task at hand.

Catalog record

CREATORYueh-Po Peng, Vincent K.M. Cheung, Li Su
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22378

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding Yueh-Po Peng, Vincent K.M. Cheung, Li Su Abstract A fundamental challenge in neuroscience is to decode mental states from brain activity. While functional magnetic resonance imaging (fMRI) offers a non-invasive approach to capture brain-wide neural dynamics with high spatial precision, decoding from fMRI data—particularly from task-evoked activity—remains challenging due to its high dimensionality, low signal-to-noise ratio, and limited within-subject data. Here, we leverage recent advances in computer vision and propose STDA-SwiFT , a transformer-based model that learns transferable representations from large-scale fMRI datasets via spatial-temporal divided attention and self-supervised contrastive learning. Using pretrained voxel-wise representations from 995 subjects in the Human Connectome Project (HCP), we show that our model substantially improves downstream decoding performance of task-evoked activity across multiple sensory and cognitive domains, even with minimal data preprocessing. We demonstrate performance gains from larger receptor fields afforded by our memory-efficient attention mechanism, as well as the impact of functional relevance in pretraining data when fine-tuning on small samples. Our work showcases transfer learning as a viable approach to harness large-scale datasets to overcome challenges in decoding brain activity from fMRI data. I Introduction The goal of brain decoding is to infer mental states from neural activity. Decoding models are not only widely employed in neuroscience research [ 1 ] , but also form the backbone of many brain-computer interfaces (BCIs) that improve the well-being of users by overcoming physical limitations, such as communication with locked-in patients [ 2 ] or facilitating stroke-recovery [ 3 ] . One popular approach for brain decoding is to use blood oxygen-level dependent (BOLD) activity recorded from functional magnetic resonance imaging (fMRI) as input features. While fMRI offers excellent (sub-)millimeter spatial resolution, it suffers several limitations: First, fMRI data is high-dimensional—often exceeding millions of voxel measurements when considering both time and space, but suffers from low signal-to-noise ratio as only task-related activity changes of \sim 1-5% are observed [ 4 ] . This presents a serious curse-of-dimensionality problem [ 5 ] and demands high computational resources. Second, the poor temporal resolution of BOLD responses ( \sim 6s between peak and stimulus onset) means that rapid changes in neural activity may be difficult to disentangle. Third, brain decoding models are hard to generalize across subjects due to individual variability in neural activity and anatomy. This is particularly the case for task fMRI as opposed to resting-state fMRI, as idiosyncratic task differences introduce additional challenges that hinder decoding transferability. Consequently, existing decoding models are typically restricted to classifying mental states from tasks specific to its training data, and rely on voxels selected from a priori -defined regions-of-interest (ROI) or aggregated activity from functionally/anatomically homogeneous regions (or brain parcels ). However, ROI- and parcellation-based decoding entail three critical disadvantages [ 6 ] : First, defining ROIs or parcels requires extensive a priori domain knowledge. This is especially problematic if the decoding task is novel and the underlying cognitive processes are not well-defined. Second, brain regions outside of selected regions may contain additional task relevant information for decoding. Third, information aggregation may involve excessive data processing and tradeoff in data granularity. These highlight the need for whole-brain -based decoding models. Recent advances in self-supervised learning (SSL) techniques have demonstrated remarkable performance in enabling models to learn representations transferable to other datasets from unlabeled data. This opens exciting novel opportunities for representation learning in fMRI data, particularly for end-to-end, whole-brain methods that could overcome issues faced in conventional ROI- and parcellation-based methods. To the best of our knowledge, no simple yet effective system for whole-brain task-fMRI decoding currently exist. To this end, we propose STDA-SWiFT , a Swin Transformer-based decoding model that effectively learns transferable features from large-scale fMRI datasets to improve performance on downstream decoding tasks. Our model operates on minimally preprocessed whole-brain fMRI images to allow for a fully end-to-end fine-tuning workflow. Inspired from recent advances in video understanding [ 7 , 8 ] , we also introduce a space-time divided attention (STDA) mechanism for the Swin Transformer architecture. This memory-efficient configuration separately models spatial and temporal dimensions to reduce computational overhead while preservING functional locality. Compared to joint-spatiotemporal (4D) attention designs, our architecture better matches the structure of fMRI data and allows for larger spatial window sizes under limited computational resources. Furthermore, under a SimCLR-based contrastive learning framework [ 9 ] , we systematically investigate the impact of different data augmentation strategies on pretraining and fine-tuning performance. While SSL techniques highly rely on data augmentation, data augmentation strategies for fMRI data is not well investigated. In fact, fMRI data differs significantly from natural images or videos: it is neither location-invariant nor scale-invariant, and exhibits complex spatiotemporal dependencies that are not well-captured by conventional augmentation strategies or architectures. We show which augmentation strategies are beneficial specifically for fMRI data. II Previous Work Here, we briefly review existing approaches to decode task-evoked brain activity from fMRI data. II-A ROI-based decoding Region of Interest (ROI)-based methods are among the earliest and most widely adopted approaches in task-based fMRI. These methods involve selecting specific anatomical or functional brain regions based on prior neuroscientific knowledge or independent localizer tasks, and analyzing neural activity within those constrained areas. By isolating brain regions known to support particular cognitive functions, ROI-based methods enable hypothesis-driven investigations with high interpretability, as well as provide a simple method for dimension reduction when building decoding models [ 10 ] . Classic studies such as Haxby et al. [ 11 ] laid the foundation for ROI-based multivoxel pattern analysis (MVPA), showing that patterns of activity within the ventral temporal cortex can discriminate between object categories such as faces and houses, even when the mean activation levels do not differ. Likewise, Haynes and Rees 2005 [ 12 ] , and Kamitani and Tong 2005 [ 13 ] demonstrated orientation biases in neurons in the early visual cortex below the conventional spatial resolution of fMRI by exploiting the spatial patterns of neighboring voxels. The searchlight method generalizes ROI-based decoding to the whole-brain by iteratively training a decoder using a small local neighborhood across all voxels [ 1 ] . Several brain computer interfaces also exploit task-related changes in target ROIs as their method of control. For example, a matrix speller by Sorger et al. [ 14 ] encodes letters from ROIs activated by motor imagery, mental calculation, or inner speech. Similarly, differential activity in the supplementary motor area when imagining playing tennis versus navigating at home has been used to demonstrate intention and communication in a vegetative-state patient [ 2 ] . Furthermore, decoded fMRI neurofeedback (DecNef), a technique where participants learn to implicitly up- or down-regulate activity of a particular ROI [ 15 ] , has been used for fear reduction training [ 16 ] and modulating facial preference [ 17 ] . In the context of large-scale datasets such as the Human Connectome Project (HCP) [ 18 ] , ROI-based analyses have been extensively applied to decode brain responses across a wide variety of tasks, including working memory, language processing, motor execution, and social cognition. For example, Barch et al. (2013) [ 18 ] used task-based fMRI data from HCP to link activation patterns in regions such as

Read the complete text