Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging
Runze Xia, Shuo Feng, Renzhi Wang, Congchi Yin, Xuyun Wen, Piji Li {xiarunze, fengshuo, rzhwang, congchiyin, wenxuyun, pjli} @nuaa.edu.cnCollege of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, ChinaMIIT Key Laboratory of Pattern Analysis and Machine Intellige · 2025
Read Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging by Runze Xia, Shuo Feng, Renzhi Wang, Congchi Yin, Xuyun Wen, Piji Li {xiarunze, fengshuo, rzhwang, congchiyin, wenxuyun, pjli} @nuaa.edu.cnCollege of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, ChinaMIIT Key Laboratory of Pattern Analysis and Machine Intellige (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22150.txt.
Introduction
Runze Xia and five co-authors present, in a 2025 paper, a way to rebuild what a person saw from their brain signals, using detailed text as a bridge. Reconstructions often lose detail or clash in meaning, which the team blame on thin semantic content in the brain data alone. Their pipeline adds a rich verbal layer between brain and image.
First, a vision-language model writes fine captions for the pictures a person viewed, and tests prove those details matter. Next, a decoder reads the brain scan and produces such captions, steered by three reward scores that grade object accuracy and text-image match. Last, the captions guide the image rebuild. The bridge can plug into existing rebuilders to sharpen them.
Catalog record
| CREATOR | Runze Xia, Shuo Feng, Renzhi Wang, Congchi Yin, Xuyun Wen, Piji Li {xiarunze, fengshuo, rzhwang, congchiyin, wenxuyun, pjli} @nuaa.edu.cnCollege of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, ChinaMIIT Key Laboratory of Pattern Analysis and Machine Intellige |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22150
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging Runze Xia, Shuo Feng, Renzhi Wang, Congchi Yin, Xuyun Wen, Piji Li {xiarunze, fengshuo, rzhwang, congchiyin, wenxuyun, pjli} @nuaa.edu.cnCollege of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, ChinaMIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, ChinaThe Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China Abstract Brain-to-Image reconstruction aims to recover visual stimuli perceived by humans from brain activity. However, the reconstructed visual stimuli often missing details and semantic inconsistencies, which may be attributed to insufficient semantic information. To address this issue, we propose an approach named Fine-grained Brain-to-Image reconstruction ( FgB2I ), which employs fine-grained text as bridge to improve image reconstruction. FgB2I comprises three key stages: detail enhancement, decoding fine-grained text descriptions, and text-bridged brain-to-image reconstruction. In the detail-enhancement stage, we leverage large vision–language models to generate fine-grained captions for visual stimuli and experimentally validate its importance. We propose three reward metrics (object accuracy, text-image semantic similarity, and image-image semantic similarity) to guide the language model in decoding fine-grained text descriptions from fMRI signals. The fine-grained text descriptions can be integrated into existing reconstruction methods to achieve fine-grained Brain-to-Image reconstruction. Keywords: Brain Decoding; fMRI; Brain-to-Image Reconstruction 1 1 footnotetext: Corresponding author. Introduction Human possesses sophisticated visual and cognitive systems that allow us to easily comprehend visual scenes and guide actions Kandel et al. (2000) ; Goodale & Milner (1992) . However, the specific mechanisms underlying this remarkable ability still remain unknown, posing a significant challenge in cognitive science, while also captivating the neuroscience community’s interest in exploring these mysteries further. The goal of Brain-to-Image reconstruction is to recover visual stimuli perceived by humans from brain activity signals. Recent advances in functional Magnetic Resonance Imaging (fMRI), particularly its high spatial resolution, have driven progress in decoding and reconstructing visual information from brain activity patterns Glover (2011) . Figure 1: The top section is the illustration of cognitive assumptions during scene observation. The bottom section demonstrates a comparison of the granularity between the detail-enhanced and the original caption. Unlike cameras that record every pixel, the human brain processes visual cognition differently Chen et al. (2023) . It concurrently handles linguistic concepts and semantic information associated with visual scenes Popham et al. (2021) ; Zhang et al. (2020) , as depicted in the top section of Figure 1 . For example, as shown in the image, when we observe a bathroom scene, our focus may be on the main objects, forming semantic concepts in the brain’s representation space such as mirror , sink , towels , and toilet . Therefore, semantics plays a vital role in Brain-to-Image reconstruction. Some studies Lu et al. (2023) ; Takagi & Nishimoto (2023) , have utilized the original image captions from the Natural Scenes Dataset (NSD) Allen et al. (2022) as semantic targets. These studies use the CLIP Radford et al. (2021) text encoder to obtain the semantic representation of image captions and train regression models to map fMRI signals to the semantic representation space, thereby achieving semantic reconstruction of images. However, in more complex visual stimuli, the simple captions often fail to adequately describe the content of the image. As a result, objects observed in the image may not appear in the semantic target, limiting the potential of semantic reconstruction. For instance, the original caption corresponding to the scene in the Figure 1 is A bathroom with a black sink and a shelf with folded towels on the rack above . This caption omits obvious objects like “toilet” and “mirror”, though information about these prominent objects is likely present in the brain activity, leading to a lack of decoding details. To address this issue, we propose detail-enhancement for image captions using large vision-language models with strong text-image comprehension capabilities to generate fine-grained captions that comprehensively capture all the main information. In the bottom section of Figure 1 , we illustrate the difference between the two types of captions, showing how the detail-enhanced caption better describes the main content of the image. Based on this, we propose a novel Brain-to-Image reconstruction method called Fine-grained Brain-to-Image ( FgB2I ), which aims to improve Brain-to-Image reconstruction by decoding fine-grained semantic descriptions from brain signals as a bridge for image reconstruction. We first train an unified brain-to-text model for all subjects. To decode consistent and fine-grained text descriptions for visual stimuli, we design three reward metrics to guide the model: object accuracy, text-image semantic similarity, and image-image semantic similarity. However, these metrics are non-differentiable and cannot be directly used for model training. Inspired by Mnih et al. (2015) , we employ the reinforce algorithm Sutton et al. (1999) to enable the model to be normally trained using these reward metrics. Through these steps, we are able to decode fine-grained textual descriptions from brain signals. We can easily integrate the decoded fine-grained textual descriptions into the high-level reconstruction of existing methods to improve Brain-to-Image reconstruction. In summary, our study makes the following contributions: • We identify the issue of missing target details in textual descriptions for Brain-to-Image reconstruction and propose the detail-enhancement stage to remedy this issue. • We introduce a novel approach named Fine-grained Brain-to-Image reconstruction ( FgB2I ), which employs detail-enhanced text as supplements for Brain-to-Image reconstruction. We use three rewards to guide the decoding of text descriptions and employ Reinforce Algorithm to address indifferentiable problems in model co-training stage. • We conduct extensive experiments on multiple existing Brain-to-Image reconstruction methods, demonstrating the necessity of detail enhancement and the effectiveness of different reward metrics in our proposed framework. Figure 2: Overview of FgB2I. (a) Detail enhancement of visual stimuli captions through LVLMs. (b) The process of decoding fine-grained text descriptions from brain signals, with the details inside the blue box further illustrated in Figure 3 . (c) The workflow for combining fine-grained text descriptions with existing methods, including semantic fusion through weighted average of text semantic embedding and the integration of text and image embeddings. Related Work Brain-to-Text Decoding Recent advances in deep learning have propelled brain decoding research, particularly in reconstructing language and visual stimuli from cognitive signals. Notable progress includes EEG2Text Wang & Ji (2022) achieving open-vocabulary sentence-level decoding from EEG, while UniCoRN Xi et al. (2023) and PredFT Yin et al. (2024) successively advanced fMRI-to-text translation through NLP-inspired architectures and predictive coding mechanisms, highlighting the growing sophistication of Brain-Computer Interfaces. Brain-to-Image Reconstruction Visual reconstruction research increasingly adopts diffusion models for their generation capabilities. Key approaches include mapping fMRI data to CLIP text features and Stable Diffusion’s VAE latent space Takagi & Nishimoto (2023) , and Ozcelik et al.’s two-stage framework Ozcelik & VanRullen (2023) combining VDVAE Child (2021) for low-level attributes with Versatile Diffusion Xu et al. (2023) guided by CLIP semantics. MindEye Scotti et al. (2023) further demonstrates how diffusion priors optimize reconstruction accuracy. Selective attention modulates the precision of neural representations during visual encoding by suppressing unattended features’ fidelity, while working memory constraints further shape object-level reconstruction through their limited capacity to maintain task-relevant signals during post-perceptual processing stages Chun & Johnson (2011) ; Luck & Ford (1998) ; Xia et al. (2024) . Method Detail Enhancement for image captions via LVLMs As shown in Figure 2 (a), we