Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI

Zheng Huang 1 Enpei Zhang 1 Yinghao Cai 1 Weikang Qiu 2 Carl Yang 3 Elynn Chen 4 Xiang Zhang 5 Rex Ying 2 Dawei Zhou 6 Yujun Yan 1 1 Dartmouth College 2 Yale University 3 Emory University 4 New York University 5 UNC Charlotte 6 Virginia Tech {zheng.huang.gr,enpei.zhang.gr,yinghao.cai,yujun.yan}@dart · 2025

Read Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI by Zheng Huang 1 Enpei Zhang 1 Yinghao Cai 1 Weikang Qiu 2 Carl Yang 3 Elynn Chen 4 Xiang Zhang 5 Rex Ying 2 Dawei Zhou 6 Yujun Yan 1 1 Dartmouth College 2 Yale University 3 Emory University 4 New York University 5 UNC Charlotte 6 Virginia Tech {zheng.huang.gr,enpei.zhang.gr,yinghao.cai,yujun.yan}@dart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16196.txt.

Introduction

Zheng Huang and nine co-authors work on reading pictures back out of brain scans. A scanner called fMRI tracks blood flow as a person views an image, and AI then tries to rebuild that image. The rebuild runs in two stages: map the scan into a code space, then let an image model draw from that code.

The team finds brain signals resemble the text space of a language model more than any image-based space. So their system, PRISM, turns scans into structured text that names objects, traits, and links between them. One module then draws each object on its own and blends them, cutting object errors. A second module picks the traits and links that best match the brain signal. On real data sets, PRISM beats earlier methods, cutting a key loss score by up to eight percent.

Catalog record

CREATORZheng Huang 1 Enpei Zhang 1 Yinghao Cai 1 Weikang Qiu 2 Carl Yang 3 Elynn Chen 4 Xiang Zhang 5 Rex Ying 2 Dawei Zhou 6 Yujun Yan 1 1 Dartmouth College 2 Yale University 3 Emory University 4 New York University 5 UNC Charlotte 6 Virginia Tech {zheng.huang.gr,enpei.zhang.gr,yinghao.cai,yujun.yan}@dart
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI Zheng Huang 1 Enpei Zhang 1 Yinghao Cai 1 Weikang Qiu 2 Carl Yang 3 Elynn Chen 4 Xiang Zhang 5 Rex Ying 2 Dawei Zhou 6 Yujun Yan 1 1 Dartmouth College 2 Yale University 3 Emory University 4 New York University 5 UNC Charlotte 6 Virginia Tech {zheng.huang.gr,enpei.zhang.gr,yinghao.cai,yujun.yan}@dartmouth.edu [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] Abstract Understanding how the brain encodes visual information is a central challenge in neuroscience and machine learning. A promising approach is to reconstruct visual stimuli—essentially images—from functional Magnetic Resonance Imaging (fMRI) signals. This involves two stages: transforming fMRI signals into a latent space and then using a pre-trained generative model to reconstruct images. The reconstruction quality depends on how similar the latent space is to the structure of neural activity and how well the generative model produces images from that space. Yet, it remains unclear which type of latent space best supports this transformation and how it should be organized to represent visual stimuli effectively. We present two key findings. First, fMRI signals are more similar to the text space of a language model than to either a vision-based space or a joint text–image space. Second, text representations and the generative model should be adapted to capture the compositional nature of visual stimuli, including objects, their detailed attributes, and relationships. Building on these insights, we propose PRISM , a model that P rojects fM R I s I gnals into a S tructured text space as an inter M ediate representation for visual stimuli reconstruction. It includes an object-centric diffusion module that generates images by composing individual objects to reduce object detection errors, and an attribute–relationship search module that automatically identifies key attributes and relationships that best aligne with the neural activity. Extensive experiments on real-world datasets demonstrate that our framework outperforms existing methods, achieving up to an 8 % 8\% reduction in perceptual loss. These results highlight the importance of using structured text as the intermediate space to bridge fMRI signals and image reconstruction. 1 Introduction Decoding visual stimuli from brain activity provides a unique lens into human perception ( Naselaris et al. 2011 ; Haufe et al. 2014 ) . A central approach uses fMRI signals—which measure neural activity through blood-oxygen-level-dependent responses—to reconstruct the images perceived by subjects ( Allen et al. 2022 ; Chang et al. 2019 ; Luo et al. 2023 ) . Recent advances in deep generative models have significantly improved these reconstructions, deepening our understanding of visual representation in the brain ( Chen et al. 2023 ) and enabling applications in brain-computer interfaces ( Sitaram et al. 2008 ) and brain-driven content generation ( Wang et al. 2024a ; Qiu et al. 2025 ) . FMRI-to-Image reconstruction involves two stages: mapping fMRI signals into a latent space and then generating images from that space. Its success depends on the similarity between the latent space and neural activity (alignment) and how well the generative model produces high-quality images. While recent studies ( Scotti et al. 2023 ; Scotti et al. 2024 ; Mai et al. 2024 ) focus on enhancing image quality using advanced generative models ( Podell et al. 2023 ; Xu et al. 2023 ) , alignment remains underexplored. Prior work often assumes that the latent space should match the modality of the stimuli, i.e., using vision model representations to reconstruct visual stimuli ( Scotti et al. 2023 ; Wang et al. 2024b ; Xia et al. 2024 ) . Some studies incorporate auxiliary semantic information from language models (LMs) ( Lin et al. 2022 ; Quan et al. 2024 ) , but still rely on vision-based representations as the core latent space. In contrast, we question whether matching the modality of visual stimuli is truly essential for reconstruction. In addition, prior work suffers from limited reconstruction quality due to a unified hidden representation that conflates objects and their attributes, often causing object detection errors, e.g., generating a tiger instead of a gray, tiger-striped cat (Appendix A ). This reflects a fundamental mismatch with human visual processing, which is object-centric and compositional rather than holistic ( Marr 1980 ; Bracci & Op de Beeck 2023 ) . Overcoming this limitation calls for generative models that explicitly capture the compositional structure of human perception. To address these issues, we propose PRISM , a model that P rojects fM R I s I gnals into a S tructured text space as an inter M ediate representation for image reconstruction. To identify the most effective intermediate space, we compare fMRI signals with representations from pre-trained vision, language, and vision–language models using established metrics ( Wang et al. 2020 ; Murphy et al. 2024 ; Keskar et al. 2016 ) . Unlike prior work assuming vision-based representations are essential, our first finding ( Section 3.1 ) shows that fMRI signals align more closely with the text space of LM, motivating the use of solely text as a bridge for reconstruction. Building on this, our second finding reveals that reconstruction quality improves when the text and the generative model are adapted to capture the compositional and relational nature of visual stimuli—encompassing objects, their attributes, and their relationships. Guided by these insights, we develop two core modules: an object-centric diffusion module that adapts the diffusion model to generate images by composing individual objects, and an attribute–relationship search module that uses a vision–language model (VLM) to automatically identify object attributes and relationships aligned with neural activity, providing structured guidance for reconstruction. Our contributions are summarized as follows: • Novel Findings: To our knowledge, we are the first to show that accurate visual stimuli reconstruction can be achieved without image-based latent representations, with LM text space effectively bridging brain activity and generative models. Furthermore, we find that adapting this text space and the generative model to capture the compositional and relational nature of visual images further improves reconstruction quality. • Novel Framework: Motivated by our empirical findings, we introduce a new fMRI-to-image reconstruction framework that adapts diffusion models for object-centric generation and leverages VLMs to automatically identify brain-aligned object attributes and relationships that can optimally guide the reconstruction. • Comprehensive Experiments: Extensive evaluations on real-world fMRI datasets demonstrate that our method achieves up to an 8\% reduction in perceptual loss compared to state-of-the-art models, highlighting the effectiveness of our framework. 2 Preliminary Notations. In our work, we denote the set of fMRI samples collected during image viewing as \mathcal{X} . Each sample is a preprocessed 1D vector \mathbf{x}_{i}\in\mathbb{R}^{v} , capturing neural activity across v voxels selected from brain regions ( Scotti et al. 2023 ) . The dataset is split into training and test subsets, with superscripts indicating the split. For example, we denote the training set with N samples as: \mathcal{X}^{\text{train}}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\} . The corresponding image stimulus for the i -th sample is denoted as: \mathbf{Y}_{i}\in\mathbb{R}^{H\times W\times 3} , which contains m objects. Problem Setup. Our goal is to reconstruct the visual images that subjects viewed during fMRI recording. Formally, we seek to learn a reconstruction function \mathcal{F}:\mathbb{R}^{v}\rightarrow\mathbb{R}^{H\times W\times 3} that maps each fMRI sample \mathbf{x}_{i} to its corresponding image stimulus \mathbf{Y}_{i} . Diffusion Model. Diffusion models ( Rombach et al. 2022 ;

Read the complete text