LaVCa: LLM-assisted Visual Cortex Captioning

Takuya Matsuyama, Shinji Nishimoto, Yu Takagi · 2025

Read LaVCa: LLM-assisted Visual Cortex Captioning by Takuya Matsuyama, Shinji Nishimoto, Yu Takagi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13606.txt.

Introduction

Takuya Matsuyama and co-authors present LaVCa, a method that writes plain language captions for what single brain voxels in the visual cortex respond to. Deep networks can predict voxel activity well, but their black box form hides what each voxel cares about. Their 2025 approach lets a large language model do the describing.

For each voxel, the method finds the images that drive it and captions their shared content. Those captions beat a prior method at matching true voxel taste. They also capture finer detail, both between voxels and within one voxel's range. The maps reveal fine splits inside known visual zones, plus voxels that encode two concepts at once.

Catalog record

CREATORTakuya Matsuyama, Shinji Nishimoto, Yu Takagi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13606

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LaVCa: LLM-assisted Visual Cortex Captioning Takuya Matsuyama, Shinji Nishimoto, Yu Takagi Abstract Understanding the property of neural populations (or voxels) in the human brain can advance our comprehension of human perceptual and cognitive processing capabilities and contribute to developing brain-inspired computer models. Recent encoding models using deep neural networks (DNNs) have successfully predicted voxel-wise activity. However, interpreting the properties that explain voxel responses remains challenging because of the black-box nature of DNNs. As a solution, we propose LLM-assisted Visual Cortex Captioning (LaVCa), a data-driven approach that uses large language models (LLMs) to generate natural-language captions for images to which voxels are selective. By applying LaVCa for image-evoked brain activity, we demonstrate that LaVCa generates captions that describe voxel selectivity more accurately than the previously proposed method. Furthermore, the captions generated by LaVCa quantitatively capture more detailed properties than the existing method at both the inter-voxel and intra-voxel levels. Furthermore, a more detailed analysis of the voxel-specific properties generated by LaVCa reveals fine-grained functional differentiation within regions of interest (ROIs) in the visual cortex and voxels that simultaneously represent multiple distinct concepts. These findings offer profound insights into human visual representations by assigning detailed captions throughout the visual cortex while highlighting the potential of LLM-based methods in understanding brain representations. Please check out our webpage at https://sites.google.com/view/lavca-llm/ . † † affiliationnotice: \icmlEqualLastAuthor Figure 1: Illustration of our paper. Our proposed method, LaVCa, generates text captions that explain voxel selectivity and surpass existing approaches, such as one-hot vectors and BrainSCUBA, enabling a more detailed description of the properties of visual cortex voxels. 1 Introduction A primary goal of computer vision is to build systems capable of processing and understanding the complex visual world in a manner akin to human perception. Studying how the human brain—with its advanced visual functions—forms its visual representations deepens our understanding of the brain’s visual network and holds promise for developing next-generation computer vision models. To understand how the human brain represents visual stimuli from the external world, researchers have used encoding models that predict voxel-level (the spatial measurement unit of fMRI) brain activity measured during the presentation of images or videos ( Naselaris et al. 2011 ) . These models typically use features derived from theoretical or data-driven hypotheses to map from stimulus properties to voxel responses ( Kay et al. 2008 ; Nishimoto et al. 2011 ; Naselaris et al. 2011 ; Huth et al. 2012 ) . Many such models use simple, interpretable features—for instance, low-level visual features generated by filters rooted in theoretical models or high-level visual features representing words using one-hot encoding—to infer voxel properties through the correlation between feature-based predictions and actual brain responses. However, because the human brain solves highly complex real-world tasks, relying solely on a small number of interpretable parameters may be insufficient ( Kriegeskorte & Douglas 2018 ) . Recent advances in deep neural networks (DNNs), which effectively handle real-world tasks through a large number of parameters, have enabled more accurate predictions of brain activity by leveraging the complex representations extracted by DNNs ( Güçlü & Van Gerven 2015 ; Schrimpf et al. 2021 ; Takagi & Nishimoto 2023 ; Denk et al. 2023 ; Antonello et al. 2024 ) . Nevertheless, because of their black-box nature, interpreting how these DNN-based features map to individual voxels remains challenging ( Abnar et al. 2019 ) . Although group-level interpretations—representing many voxels in terms of a small set of universal, interpretable axes—are possible ( Huth et al. 2016 ; Lescroart & Gallant 2019 ; Nakagi et al. 2024 ) , voxel-level interpretation is critical for exploring more nuanced aspects of brain representations. In this study, we address the difficulty of voxel-level interpretation with a new method called LLM-assisted Visual Cortex Captioning ( LaVCa ), which generates data-driven captions for individual voxels (Figure 1 ). LaVCa proceeds in four steps: (1) building voxel-wise encoding models for brain activity evoked by images, (2) identifying the optimal images for each voxel’s encoding model using an augmented image dataset (3) generating captions for these optimal images, and (4) creating concise summaries from those captions. By leveraging large language models (LLMs) with access to a vast, open-ended vocabulary, LaVCa generates diverse inter-voxel captions. Moreover, generating captions from multiple keywords enables us to capture diverse intra-voxel properties. Our contributions are as follows: 1. We propose LLM-assisted Visual Cortex Captioning (LaVCa), a data-driven technique that leverages LLMs to generate natural language captions of voxel-level visual selectivity. 2. We demonstrate that LaVCa produces more accurate captions than the earlier method BrainSCUBA ( Luo et al. 2023 ) and better characterizes voxel-wise visual selectivity through brain activity prediction. 3. We also demonstrate that LaVCa can generate highly interpretable and accurate captions without sacrificing information from the optimal images (Figure 2 ). 4. The captions generated by LaVCa quantitatively capture more detailed properties than the existing method at both the inter-voxel and intra-voxel levels. 5. More detailed analysis of the voxel-specific properties generated by LaVCa reveals fine-grained functional differentiation within regions of interest (ROIs) in the visual cortex and voxels that simultaneously represent multiple distinct concepts. Figure 2: The relationship between sentence-level prediction performance and the number of words in voxel captions for a single subject (subj01). The number following “LaVCa” indicates the number of optimal images used for summarization, while the number following “Concat” indicates the number of concatenated captions from the optimal images. Error bars indicate the standard error. LaVCa explains the properties of voxels well using a small number of words. 2 Related Work Encoding models have been widely used in neuroscience to interpret neural representations ( Kay et al. 2008 ; Nishimoto et al. 2011 ; Naselaris et al. 2011 ; Huth et al. 2012 ) . These studies typically use interpretable low-level visual features and high-level semantic features, such as one-hot encodings, permitting the interpretation of individual voxels. Encoding models that rely on features extracted from DNNs can explain brain activity more effectively than those using simpler, more interpretable features ( Güçlü & Van Gerven 2015 ; Schrimpf et al. 2021 ; Takagi & Nishimoto 2023 ; Antonello et al. 2024 ) ; however, their complexity complicates voxel-level interpretation. Consequently, several methods aggregate all voxels into a limited number of universal and interpretable axes ( Huth et al. 2016 ; Lescroart & Gallant 2019 ; Nakagi et al. 2024 ) . More recently, to enable finer analyses of neural representations at the voxel level, data-driven approaches have been proposed to generate natural-language descriptions of voxel selectivity ( Luo et al. 2023 ; Singh et al. 2023 ) . BrainSCUBA ( Luo et al. 2023 ) , for example, is an end-to-end method that uses an existing image captioning model to produce voxel-wise captions for the visual cortex. Similarly, SASC ( Singh et al. 2023 ) uses an fMRI dataset recorded during speech listening ( LeBel et al. 2023 ) , uses an LLM to merge multiple short phrases—those with the highest predicted voxel activations—into a single, data-driven caption, thus describing the semantic properties of voxels. Our proposed method also generates data-driven voxel captions but differs in several ways. First, BrainSCUBA is constrained to pre-existing, end-to-end image captioning models. In contrast, our approach divides the task

Read the complete text