Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representations
Christos Zangos, Danish Ebadulla, Thomas Sprague, Ambuj Singh · 2025
Read Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representations by Christos Zangos, Danish Ebadulla, Thomas Sprague, Ambuj Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01670.txt.
Introduction
Rebuilding what a person saw from their brain scans is a flagship problem in neuroscience, but each new subject brings scarce data. Christos Zangos and co-authors propose a training recipe in 2025 that adapts a pre-trained reconstruction model to a new person with very little scanning.
First, small subject-specific adapters are explicitly aligned to a reference subject using a shared set of calibration images. Then the whole pipeline is fine-tuned while that alignment is kept. The shared space that results is not just common but semantically matched across people. With only 24 minutes of new data, the method beat baselines fine-tuned on a full hour. Gains held across model types, reference subjects, and data sets.
Catalog record
| CREATOR | Christos Zangos, Danish Ebadulla, Thomas Sprague, Ambuj Singh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01670
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representations Christos Zangos, Danish Ebadulla, Thomas Sprague, Ambuj Singh Abstract This work introduces a novel approach to fMRI-based visual image reconstruction using a subject-agnostic common representation space. We show that the brain signals of the subjects can be aligned in this common space during training to form a semantically aligned common brain. This is leveraged to demonstrate that aligning subject-specific lightweight modules to a reference subject is significantly more efficient than traditional end-to-end training methods. Our approach excels in low-data scenarios. We evaluate our methods on different datasets, demonstrating that the common space is subject and dataset agnostic. † † affiliationnotice: Equal contribution 1 Introduction Over the past several decades, the use of machine learning techniques has enabled decoding and/or reconstruction of information represented in neural activation patterns measured in awake, behaving humans. Because the neural representations supporting behavior likely entail patterns of activity across large swaths of tissue, these approaches are thought to offer a key step towards characterizing and quantifying the computational principles brains use to support cognition ( Naselaris et al. 2011 ) . Early approaches, including self-supervised learning frameworks ( Beliy et al. 2019 ; Gaziv et al. 2022 ; Mozafari et al. 2020 ; Shen et al. 2019 ; Seeliger et al. 2018 ; Ren et al. 2021 ) struggled to generate semantically accurate image reconstructions, often resulting in images with limited resemblance to the original stimuli. Furthermore, these methods required large amounts of data for training, and their generalization to a wider population of subjects was severely limited. Significant advancements were achieved by incorporating multimodal data, such as text ( Lin et al. 2022 ) , and later by leveraging the generative power of latent diffusion models (LDMs) ( Takagi & Nishimoto 2023 ; Scotti et al. 2023 ; Lu et al. 2023 ; Xia et al. 2024 ) . These models introduced novel mapping techniques to project fMRI signals into the input space of LDMs, yielding higher-quality natural images. However, despite the improved fidelity, the issue of generalization to new participants persisted. In response, ( Scotti et al. 2024 ) proposed projecting fMRI data into a shared representation space, demonstrating significant improvements in both performance and generalizability. Beyond being able to use decoding and reconstruction approaches to provide evidence of the existence of information represented in a particular brain activation pattern, an additional goal is to compare decoding or reconstruction performance across experimental manipulations, such as cued visual attention ( Serences & Boynton 2007 ; Kamitani & Tong 2005 ; Scolari et al. 2012 ; Sprague & Serences 2013 ; Itthipuripat et al. 2019 ; Sprague et al. 2018 ) and visual working memory ( Serences et al. 2009 ; Harrison & Tong 2009 ; Ester et al. 2013 ; Sprague et al. 2014 ; Christophel et al. 2012 ; Li et al. 2021 ) Ḣowever, for these approaches to be useful for a cognitive neuroscience lab, the lab would need to acquire sufficient data to train a decoding or reconstruction model for each individual participant (often >20 per study to attain statistical power). The state-of-the-art methods described above use extremely massive datasets with dozens of hours of fMRI data per participant, which is not feasible in almost all cases. Thus, there is a clear need to maximize the efficiency and generalizability of image reconstruction methods such that a model for a new participant can be trained using as little data as possible. In this work, we introduce an efficient and generalizable approach to fMRI-based visual reconstruction that leverages a subject-agnostic common representation space. Unlike previous approaches that perform end-to-end training of fMRI signals to mapped contextual embeddings, we propose aligning brain activity from new subjects to a pre-trained reference subject in a shared representation space using Adapter Alignment (AA) training. Throughout this work, we refer to lightweight subject-specific modules designed to map input fMRI signals to a shared representation space as ”adapters” ( Liu et al. 2024 ) . These modules are implemented as single-layer neural networks. In their linear form, adapters consist of a single linear transformation layer, while in the non-linear form, they include a linear layer followed by a non-linear activation function such as GELU ( Hendrycks & Gimpel 2023 ) . Using AA training, we significantly reduce the amount of fMRI data required and the training time necessary to achieve accurate reconstructions and improve the reconstruction quality. Our method achieves better performance than the training method used by existing state-of-the-art models, but with far fewer training hours, making it feasible for cognitive neuroscience labs with limited data collection resources. We build on this approach by also proposing an adaptive image selection strategy, allowing for optimal fine-tuning of new subjects with minimal data, further improving the practicality and accessibility of fMRI-based image reconstruction. Finally, we evaluate our proposed methods on different datasets. Our contributions to this work are as follows: • We provide strong evidence for the existence of a common visual representation space across subjects, where brain activity from different individuals can be aligned in a shared space. By extracting and analyzing embeddings from a pre-trained model, we show that even without explicit alignment, subject-specific fMRI signals, when mapped to a common space display similar behavior and representation structures. Furthermore, we demonstrate that applying a simple orthogonal transformation improves the alignment of embeddings between subjects, and using non-linear adapters with explicit mapping leads to almost perfect alignment across subjects in the common space. (Section 2 ) • We introduce Adapter Alignment (AA), a novel approach to facilitate efficient subject-specific alignment in fMRI-based visual reconstruction. Our training pipeline first pre-trains on a reference subject, then aligns new subjects’ brain data using lightweight adapters that map them to the shared common space. This process, which is more efficient than traditional end-to-end training, enables faster convergence and better performance, particularly in low-data scenarios. By incorporating MSE losses at the adapter level for common images, we show that our method not only reduces training time but also improves generalization and reconstruction quality. We also present scaling experiments to show that AA training is scalable and agnostic to the components used in the pipeline. (Section 3 , Appendix Section A ) • We develop a greedy image selection algorithm that identifies the most representative subset of images for training new subjects. By efficiently selecting fMRI signals that capture the most informative aspects of the common space, this strategy significantly reduces the amount of data required for fine-tuning without sacrificing performance (from a baseline of 250 images to 150 images). This makes our method more practical for real-world applications, where acquiring large fMRI datasets is often prohibitively expensive. (Section 4 ) • We validate the generalizability of our approach by applying it to data acquired from different MRI scanners. In particular, we leverage the THINGS dataset ( Hebart et al. 2023 ) to fine-tune our pre-trained models, originally trained on the NSD dataset. Despite the variations in scanning protocols and data characteristics, our Adaptive Alignment (AA) strategy continues to outperform conventional fine-tuning. (Section 5 ) 2 A Common Visual Representation Space Central to our approach is the hypothesis that each individual’s brain (or at least