Versatile Framework with Semantic and Structural guidance for Image Reconstruction from Brain Activity

Yizhuo Lu, Changde Du, Qiongyi Zhou, Liuyun Jiang, Huiguang He * · 2026

Read Versatile Framework with Semantic and Structural guidance for Image Reconstruction from Brain Activity by Yizhuo Lu, Changde Du, Qiongyi Zhou, Liuyun Jiang, Huiguang He * (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00121.txt.

Introduction

Lu and co-authors present MindDiffuser, a way to rebuild a seen image from brain waves. The task matters for brain links that let minds steer machines. Past tools got the gist of a scene right. Yet they lost fine points like place, tilt, and size.

MindDiffuser works in two steps. First, word codes read from the brain feed a text-to-image model. That gives a rough draft with the right themes. Next, flat sight codes steer small fixes by back steps. The loop pulls shape and place toward the true scene. Tests span three kinds of brain scans. The authors report clear gains over past top tools on all three.

Catalog record

CREATORYizhuo Lu, Changde Du, Qiongyi Zhou, Liuyun Jiang, Huiguang He *
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00121

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Versatile Framework with Semantic and Structural guidance for Image Reconstruction from Brain Activity Yizhuo Lu, Changde Du, Qiongyi Zhou, Liuyun Jiang, Huiguang He * Abstract Reconstructing visual stimuli from brain recordings has been a meaningful and challenging task in brain decoding. Especially, the achievement of precise and controllable image reconstruction bears great significance in propelling the progress and utilization of brain-computer interfaces. Recent methods, leveraging advances in the power of text-to-image generation models, have reconstructed images that closely approximate complex natural stimuli in terms of semantics (e.g., concepts and objects). However, they struggle to maintain consistency with the original stimuli in fine-grained structural information (e.g., position, orientation and size), which undermines both the controllability and interpretability of the models. To address the aforementioned issues, we propose a two-stage image reconstruction framework, termed MindDiffuser. In Stage 1, Contrastive Language-Image Pretraining (CLIP) text embeddings decoded from brain responses are input into Stable Diffusion, generating a preliminary image containing semantic information. In Stage 2, we use decoded shallow CLIP visual features as supervisory signals, iteratively refining the feature vectors from Stage 1 via backpropagation to align structural information. We conducted extensive experiments on brain response datasets across three modalities (fMRI, EEG, MEG) elicited by visual stimuli, demonstrating that our framework significantly enhances the performance of previous state-of-the-art models, highlighting the effectiveness and versatility of our approach. Spatial and temporal visualization results further support the neurobiological plausibility of our framework, providing guidance for future neural decoding efforts across different brain signal modalities. 2025 † † footnotetext: * Corresponding author. Email: [email protected] keywords Brain decoding, controlled image reconstruction, versatile framework, diffusion model. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Figure 1: An illustration of the forward and reverse pathways in visual processing systems. Forward visual processing in the human brain involves the ventral stream, which processes semantic-level information like objects and concepts, and the dorsal stream, which processes structural-level spatial information such as position and orientation. Our framework reverses these two pathways to decode both semantic and structural features, enabling accurate reconstruction of stimulus images. The human visual system excels at perceiving and interpreting complex visual stimuli with remarkable efficiency and robustness, far exceeding current AI models. Understanding the neural mechanisms behind these processes is crucial for unraveling brain function Rakhimberdina et al. (2021) ; Zhou et al. (2022) , despite the challenges. A central focus of this research is neural decoding Huang et al. (2023) ; Zhou et al. (2025) , which seeks to link visual stimuli with brain recordings. Neural decoding techniques encompass classification, identification, and reconstruction tasks, with this study emphasizing reconstruction-the most challenging among them. Early image reconstruction studies predominantly employed end-to-end paradigms for training Du et al. (2020) ; Du et al. (2018) ; Shen et al. (2019a) ; Beliy et al. (2019) . However, due to the scarcity of paired image-stimulus and brain-response data in this field, these paradigms often yielded blurry reconstructions that lacked precise semantic content. Recent advancements in large-scale text-to-image (T2I) models, such as BigGAN Donahue and Simonyan (2019) and Stable Diffusion Rombach et al. (2022) , have led to significant progress in the field. Trained on extensive image-text pair datasets, these models have acquired rich priors that enable them to generate highly realistic images based on textual prompts. This rapid development has garnered considerable attention from researchers in neuroscience. Many studies first map fMRI data to the corresponding multimodal representations of stimulus images, and then input these representations into pre-trained T2I models. By leveraging the T2I model’s powerful visual priors, they have successfully reconstructed semantically coherent natural images Lin et al. (2022) ; Ozcelik and VanRullen (2023) ; Takagi and Nishimoto (2023) ; Gu et al. (2022) ; Scotti et al. (2024) . Although this paradigm offers advantages such as lower training difficulty and reconstructions with clear semantic content, it also has notable drawbacks. Additionally, due to the lack of additional constraints on the T2I model, the reconstructed images fail to align with the original images in terms of structural information, including size, position, and shape. To address this issue, we draw inspiration from the visual cortex processing mechanisms of the brain. As shown in Fig. 1 , in the human brain, forward visual information processing primarily relies on two pathways: the ventral stream and the dorsal stream. The ventral stream is responsible for processing semantic-level information such as concepts and objects, while the dorsal stream handles structural-level spatial information, including position and orientation. This collaboration enables humans to comprehend both what is present in an image and where it is located Vaziri-Pashkam and Xu (2017) ; Zachariou et al. (2014) . It is evident that prior work within this framework has only reversed the ventral stream, neglecting the structural-level spatial information processed by the dorsal stream, which has led to suboptimal reconstruction outcomes. In this work, we propose MindDiffuser, a two-stage image reconstruction framework that reverses both the ventral and dorsal streams. In Stage 1, CLIP text embeddings decoded from brain responses are input into Stable Diffusion, generating a preliminary image that captures the semantic content, thus providing an initial understanding of “ what is in the image.” In Stage 2, we use decoded shallow CLIP visual features as supervisory signals, iteratively refining the feature vectors from Stage 1 via backpropagation to align structural information, thereby constraining “ where the objects are located in the image.” Note that there is a previous conference version 1 1 1 Code are available at: https://github.com/ReedOnePeck/MindDiffuser of this work Lu et al. (2023) . In contrast to previous work, we extend our method from fMRI to EEG and MEG data, validating our framework on ten top models from various modalities and architectures. This version also includes extensive experimental and temporal/spatial interpretability analyses. Our contributions are summarized as follows: (1) Controllable framework: Inspired by the visual cortex’s information processing mechanisms, we propose a two-stage framework (Fig. 2 ) that reverses the ventral and dorsal pathways, guided by semantic and structural information, addressing the limitation of previous methods that fail to incorporate both aspects. (2) Versatile performance: We conducted extensive experiments on datasets from different modalities (fMRI, EEG and MEG) and reconstruction methods based on various generative models (GAN, Diffusion). The results demonstrate that our framework can be integrated into existing models to significantly enhance their low-level structural metrics, with minimal impact on high-level semantic metrics (Table 4 ). (3) Interpretability: Considering the characteristics of different modality data, we conducted detailed spatial and temporal visualization analyses, revealing the interpretability and rationality of the framework in relation to the experimental results (Fig. 9 ). 2 Related Work 2.1 Generative models for neural decoding Early image generation models primarily focused on VAE Kingma and Welling (2014) and GAN Goodfellow et al. (2020) , which are lightweight but difficult to scale up. Recently, diffusion models Wijmans and Baker (1995) ; Ho et al. (2020) have rapidly emerged as a new generative paradigm. In these models, Gaussian noise is added to an image during the forward diffusion process until collapse, followed by a reverse denoising process to generate the image. With strong multimodal representations and large-scale text-image datasets, DALLE-2

Read the complete text