A Survey of fMRI to Image Reconstruction

Weiyu Guo, Guoying Sun, Jianxiang He, Tong Shao, Shaoguang Wang, Ziyang Chen, Meisheng Hong, Ying Sun, Hui Xiong · 2025

Read A Survey of fMRI to Image Reconstruction by Weiyu Guo, Guoying Sun, Jianxiang He, Tong Shao, Shaoguang Wang, Ziyang Chen, Meisheng Hong, Ying Sun, Hui Xiong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16861.txt.

Introduction

Weiyu Guo and eight co-authors survey the young field of rebuilding pictures from brain scans. Given a person's functional MRI while they view images, models now guess the image, a striking window on perception. Their 2025 review names the task fMRI-to-image learning and organizes it end to end: encoding the brain signal, mapping features, and generating the picture. Three stubborn problems head the list: too little data, brains that differ across people, and outputs that miss the meaning of the scene.

A first systematic map helps newcomers see the whole pipeline and its weak joints. The uses reach from basic neuroscience to brain-computer interfaces. The authors sort methods by their stage in the pipeline. Their review also flags where progress is real and where it is demo-deep. The field's promise, reading perception, depends on fixing the data problem.

Catalog record

CREATORWeiyu Guo, Guoying Sun, Jianxiang He, Tong Shao, Shaoguang Wang, Ziyang Chen, Meisheng Hong, Ying Sun, Hui Xiong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16861

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Survey of fMRI to Image Reconstruction Weiyu Guo, Guoying Sun, Jianxiang He, Tong Shao, Shaoguang Wang, Ziyang Chen, Meisheng Hong, Ying Sun, Hui Xiong Abstract Functional magnetic resonance imaging (fMRI)-based image reconstruction plays a pivotal role in decoding human perception, with applications in neuroscience and brain-computer interfaces. While recent advancements in deep learning and large-scale datasets have driven progress, challenges such as data scarcity, cross-subject variability, and low semantic consistency persist. To address these issues, we introduce the concept of fMRI-to-Image Learning (fMRI2Image) and present the first systematic review in this field. This review highlights key challenges, categorizes methodologies such as fMRI signal encoding, feature mapping, and image generator. Finally, promising research directions are proposed to advance this emerging frontier, providing a reference for future studies. 1 Introduction Functional magnetic resonance imaging (fMRI) is a powerful neuroimaging technique that measures brain activity indirectly by detecting changes in blood oxygen levels, which reflect neuronal activity. Recently, with the rise of deep learning models like Contrastive Language-Image Pre-training (CLIP) and Latent Diffusion Models (LDMs), along with the availability of large-scale fMRI datasets such as the Natural Scenes Dataset (NSD) Allen and St-Yves 2021 , reconstructing visual perception from fMRI signals has become an exciting area of research. This approach not only enhances our understanding of how the brain encodes visual information but also opens new possibilities for applications like brain-computer interfaces (BCIs). By converting fMRI data into interpretable visual forms, we can explore the brain’s internal representations and gain deeper insights into human perception and cognition. Reconstructing visual images from fMRI signals presents several significant challenges, both in terms of data and modeling. Data-related challenges include the inherent complexity and variability of fMRI data, where individual differences in brain activity patterns can result in substantial variation across subjects. This variability complicates the creation of robust, generalized models. Moreover, fMRI datasets are often limited in diversity, particularly in terms of the number of subjects, which hinders the training of cross-subjects models. Another key issue is the misalignment of data across subjects, arising from the inherent variability in input dimensions due to differences in brain size. On the modeling side, mapping fMRI signals to the high-dimensional image space remains a challenging task. Effectively translating complex and noisy brain data into coherent visual representations requires advanced techniques that can capture subtle nuances in neural activity. Furthermore, while progress has been made in generating visual content from brain signals, issues with image quality and low-level semantic consistency persist. Current models still struggle to consistently generate images that accurately represent both fine details and the broader context of the visual stimuli. Additionally, traditional approaches often suffer from overfitting, particularly when trained on small datasets, as the models tend to memorize rather than generalize across different subjects and brain patterns. These challenges underscore the need for more sophisticated techniques to bridge the gap between fMRI and visual reconstructions. To address the challenges of fMRI-to-image reconstruction, current methodologies are typically organized into three stages: fMRI signal encoding, feature mapping, and image reconstruction. Due to the varying dimensionalities and the presence of substantial noise in fMRI data across different subjects, many approaches convert fMRI data into one-dimensional representations of different lengths to reduce noise. Recently some methods, preserve spatial correlations by transforming the data into a two-dimensional standard brain map. One-dimensional data is typically processed using MLPs or transformers, while two-dimensional brain maps are often processed using CNNs or ViTs. Additionally, pretrained models on large neuroimaging datasets and techniques like Masked Autoencoders (MAE) further improve feature extraction, minimizing the need for paired fMRI-image datasets. The feature mapping stage aligns fMRI features with visual content by mapping features from different brain regions to corresponding embeddings. For instance, features from language-related brain areas are mapped to CLIP’s text embeddings. Finally, the image reconstruction stage generates visual content based on these aligned features. While earlier approaches struggled with low semantic consistency, recent advancements in diffusion models and Latent Diffusion Models (LDMs) like Stable Diffusion have significantly improved image quality and computational efficiency. Fine-tuning these models on specific fMRI datasets further addresses data scarcity and improves model generalization, highlighting the unique challenges of fMRI-to-image reconstruction compared to traditional text-to-image generation. This review focuses on the latest advancements in reconstructing visual images from fMRI signals, a rapidly evolving interdisciplinary research area. It systematically organizes and classifies recent studies based on their methodologies and optimization objectives, while also highlighting the latest publicly available datasets that have facilitated progress in this field. 2 Datasets The fMRI-to-image (or-video) dataset facilitates the investigation of brain responses to static and dynamic visual stimuli. Data collection typically involves recruiting qualified participants from university communities. During experiments, participants perform continuous recognition tasks, viewing images of natural scenes or movie clips and indicating whether each stimulus has been previously encountered Allen and St-Yves 2021 . The images are sourced from specific databases, while movie clips are selected from designated video databases Nishimoto and Vu 2011 . High-field-strength fMRI scanners are employed to record participants’ neural activity in response to visual stimuli. Table 1 provides an overview of the datasets, categorized by type, image/video source, number of subjects, sample size, and fMRI device specifications. Dataset Year Type Resource Subject Sample Device Vim-1 2008 Image Corel Stock Photo Libraries/Berkeley Segmentation Dataset 2 2*1750/120 4 T INOVA MR scanner/Quadrature transmit-receive surface coil GOD 2015 Image ImageNet 5 5*1200/50 3.0-Tesla Siemens MAGNETOM Verio EEG-VOA 2016 Image ImageNet 6 6*1600/400 actiCAP-128Ch2/Brainvision DAQs BOLD 2018 Image ImageNet/SUN/COCO 4 4*5254 3.0-Tesla Siemens MAGNETOM Verio DIR 2017 Image ImageNet 3 3*1200/100 3.0-Tesla Siemens MAGNETOM Verio Faces 2018 Image CelebA 4 4*8000/20 3T Philips ACHIEVA scanner largeEEG 2022 Image THINGS 10 82160 64-channel EASYCAP OCD 2020 Image ImageNet 5 5*2250/500 3T Prismafit scanner THINGS-data 2023 Image THINGS 3 3*8740 Siemens 3T MAGNETOM Prisma/CTF 275 MEG system NOD 2023 Image ImageNet/COCO 30 57120 Siemens 3T MAGNETOM Prisma NSD 2021 Image COCO 8 8*9000/1000 7T Siemens Magnetom 48 passively-shielded scanner/single-channel-transmit 32-channel-receive RF head coil VER 2011 Video Natural movies 3 3*2400/180s 4T Varian INOVA scanner DNV 2016 Video Natural movies 3 3*972p 3T MRI system/16-channel receive-only phase-array surface coil STNS 2019 Video Natural movies 1 1*23h Siemens 3T MAGNETOM Prisma TGBH 2020 Video The Grand Budapest Hotel 25 25*2h Siemens 3T MAGNETOM Prisma NHA 2022 Video Natural recording 4 4*300p 3T Siements TimTrio MR scanner/32-channel phase array head coil NATVIEW-EEGfMRI 2023 Video Checkerboard stimulus/Short film 22 22*5958s 3T Siements TimTrio MR scanner/MR-compatible system by Brain Products BMD 2024 Video Moments in Time 10 10*1000p 3T Trio Siemens scanner/32-channel head coil m-fMRI 2024 Video TSA2/UNBC-McMaster/Ganis & Kievit/Polti 101 101*6h Siemens 3T MAGNETOM Prisma NFED 2024 Video DFEW/CAER 5 5*1320p Siemens 3T MAGNETOM Prisma Table 1: Overview of Datasets. In the Vim-1 dataset, for instance, the notation [2*1750/120] denotes that six participants viewed 1750 train images and 120 test images, yielding a total of 6*1870 samples. The absence of the / symbol indicates that no training-test split has been performed. The absence of the * indicates that the experimental conditions were different for each participant. For image datasets, the sample unit is the number of images. For video datasets, s denotes seconds, h denotes hours, and p denotes the number of videos. 2.1 fMRI to

Read the complete text