Optimized two-stage AI-based Neural Decoding for Enhanced Visual Stimulus Reconstruction from fMRI Data
Lorenzo Veronese, Andrea Moglia, Luca Mainardi, Pietro Cerveri · 2024
Read Optimized two-stage AI-based Neural Decoding for Enhanced Visual Stimulus Reconstruction from fMRI Data by Lorenzo Veronese, Andrea Moglia, Luca Mainardi, Pietro Cerveri (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13237.txt.
Introduction
Lorenzo Veronese, Andrea Moglia, Luca Mainardi, and Pietro Cerveri work on reading pictures from brain scans, in a paper published in 2024. Their tools rebuild what a person saw from signals in a scanner, using image generators guided by brain data. The data are noisy and complex, so newer systems split the task into a rough first image and a polish step.
The team swap the usual straight-line first step for a deep network that reshapes the brain signal more flexibly. On a public set of scenes viewed in a scanner, their images matched the true pictures about two percent better in structure and about four percent better in meaning. Tests also showed the first step carried the structure, while noise hurt looks more than meaning. Better decoding may aid study of vision itself.
Catalog record
| CREATOR | Lorenzo Veronese, Andrea Moglia, Luca Mainardi, Pietro Cerveri |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimized two-stage AI-based Neural Decoding for Enhanced Visual Stimulus Reconstruction from fMRI Data Lorenzo Veronese, Andrea Moglia, Luca Mainardi, Pietro Cerveri Abstract AI-based neural decoding reconstructs visual perception by leveraging generative models to map brain activity, measured through functional MRI (fMRI), into latent hierarchical representations. Traditionally, ridge linear models transform fMRI into a latent space, which is then decoded using latent diffusion models (LDM) via a pre-trained variational autoencoder (VAE). Due to the complexity and noisiness of fMRI data, newer approaches split the reconstruction into two sequential steps, the first one providing a rough visual approximation, the second on improving the stimulus prediction via LDM endowed by CLIP embeddings. This work proposes a non-linear deep network to improve fMRI latent space representation, optimizing the dimensionality alike. Experiments on the Natural Scenes Dataset showed that the proposed architecture improved the structural similarity of the reconstructed image by about 2% with respect to the state-of-the-art model, based on ridge linear transform. The reconstructed image’s semantics improved by about 4%, measured by perceptual similarity, with respect to the state-of-the-art. The noise sensitivity analysis of the LDM showed that the role of the first stage was fundamental to predict the stimulus featuring high structural similarity. Conversely, providing a large noise stimulus affected less the semantics of the predicted stimulus, while the structural similarity between the ground truth and predicted stimulus was very poor. The findings underscore the importance of leveraging non-linear relationships between BOLD signal and the latent representation and two-stage generative AI for optimizing the fidelity of reconstructed visual stimuli from noisy fMRI data. A Preprint Keywords Neural decoding \cdot MRI imagery \cdot Generative Artificial Intelligence 1 Introduction Understanding the intricate relationship between neural activity and visual perception is one of the central aims in neuroscience research [ 11 , 41 ] . Functional magnetic resonance imaging (fMRI) has been described to offer a window into such a relationship by capturing blood-oxygen-level-dependent (BOLD) signals that indirectly reflect neuron functional patterns [ 5 ] . However, the complex and noisy nature of fMRI data presents a significant challenge in reconstructing the underlying visual stimuli. Likewise, significant barriers to accurate reconstruction have been described to emerge when dealing with a scene conveying rich semantic meaning, compared to reconstructing low-complexity content [ 18 ] . Leveraging the power of deep neural networks (DNNs), novel image decoding approaches have been pioneering to shed light on how humans learn to decode and interpret visual information [ 41 , 31 ] . For instance, deep networks have been very recently investigated to emulate perceptual systems for tasks like object recognition [ 13 ] , mimic gradual learning in categorization taksk [ 6 ] , and even simulate how early visual abilities can adapt as the brain matures [ 16 ] . In the field of multiple hierarchical representations, based on DNNs, the underlying idea is that the BOLD signal (multi-voxel fMRI activation pattern) is implicitly conveying latent information. This information can be decoded using a similar approach to how features are extracted from images in deep convolutional neural networks, like the well-known ImageNet model [ 31 ] . The BOLD signal is typically preprocessed using General Linear Model (GLM) analysis, which generates single-trial response estimates, known as betas, for each voxel in response to individual stimuli. Subsequently, the relationship between the beta weights and the derived latent space is established using techniques such as ridge linear regression [ 19 ] . The relationship between the original fMRI BOLD space, which contains the raw activation patterns across voxels (brain regions), and the derived latent spaces is established through techniques like ridge linear regression [ 19 ] . This method builds upon ordinary least squares regression, but with an added regularization term that helps prevent overfitting [ 7 ] . By applying ridge regression (or similar techniques), the complex BOLD signal is essentially compressed into a lower-dimensional latent space. This compressed signal then becomes the key for decoding. Various deep learning architectures, like Variational Autoencoders (VAE) [ 33 ] , Generative Adversarial Networks (GAN) [ 37 ] , and more recently Latent Diffusion Models (LDM) [ 35 , 8 ] , have been employed for this purpose. While spatially related to the original visual stimulus, the reconstruction from the BOLD latent signal was a very coarse low-resolution image [ 31 ] . Increasing the size of the latent space up to 30 layers was documented to be very computationally demanding [ 21 ] , providing nonetheless questionable advantage in the reconstruction quality [ 30 ] . In order to empower the reconstruction ability of the visual stimulus, a second processing stage has been recently proposed by embedding Contrastive Language-Image Pre-training (CLIP) models [ 14 , 17 , 9 ] . CLIP models are pre-trained on a massive dataset (400 million) of image-text pairs available on the Internet, allowing them to learn the relationship between visual and language features. Hence using CLIP, decoding models exploit concurrently BOLD and text latent representations. This enables the link of specific brain activity patterns to visual elements described in the text, which has been documented to improve the reconstruction quality [ 30 , 21 ] . In order to enable the use of such information, the relation between CLIP space and BOLD latent space must be learned, for instance using ridge linear regression models as in the first stage. However, how output uncertainty of the first processing stage affected the second stage was not systematically investigated, posing questions about the real utility of the first processing stage. In this work, we mainly focused on three main shortcomings, namely the linear mapping of the BOLD signal to the latent space, the large dimensional VAE space, and the sensitivity of the final reconstruction to the uncertainty of the first stage output. Despite the traditional ridge linear regression, we here proposed to map the BOLD signal in the latent representation by means of a non-linear neural network. In order to cope with the computational demands of the VAE dimensionality, an ablation analysis was carried out leading to optimizing the BOLD encoding size. The effectiveness of the improvements and analyses was tested using the Natural Scenes Dataset (NSD) [ 1 ] acquired with 7T fMRI measurements on eight healthy subjects while they visualized natural color images from the Common Objects in Context (COCO) dataset, composed of 73,000 images [ 15 ] . Our main contributions can be thus synthesizes as follows: • modeling latent space of beta weights using gated-recurrent-unit (GRU) network, which was tested against other deep architecture and state-of-the-art linear ridge regressor; • optimization of the latent space size with respect to output quality and computational cost; • noise sensitivity analysis to increase the understanding of the first stage role in the final stimulus reconstruction. 2 Related Works 2.1 Multimodal Contrastive Learning Pre-training Models The CLIP model [ 28 ] is a special multi-modal neural network devoted to building a shared embedding space linking images and textual descriptions. A CLIP model is usually pre-trained on a very large amount of image-text pairs, using contrastive loss functions [ 42 ] . For instance, the symmetric cross-entropy loss maximizes the cosine similarity of image-text pairs that are true matches and