Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
Subba Reddy Oota, Akshett Jindal, Ishani Mondal, Khushbu Pahwa, Satya Sai Srinath Namburi, Manish Shrivastava, Maneesh Singh, Bapi S. Raju, Manish Gupta · 2025
Read Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain) by Subba Reddy Oota, Akshett Jindal, Ishani Mondal, Khushbu Pahwa, Satya Sai Srinath Namburi, Manish Shrivastava, Maneesh Singh, Bapi S. Raju, Manish Gupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20029.txt.
Introduction
Nine authors ask whether teaching multimodal models to follow instructions brings their inner patterns closer to the brain's. Earlier work found that language models, never trained on brain data, still predict neural activity well. The new instruction-tuned multimodal models answer open-ended visual tasks, but nobody knew whether following spoken-style instructions improved their match to vision areas. The team fed model answers to ten different instructions, given while people watched natural scenes, into brain encoding models.
The multimodal models predicted visual brain activity far better than vision-only models, and about as well as a strong non-instruction model. Not every instruction helped the match, though. Counting and recognition instructions lined up best with brain signals. Much of the explained activity was shared across instructions, which suggests the models still blur task-specific detail.
Catalog record
| CREATOR | Subba Reddy Oota, Akshett Jindal, Ishani Mondal, Khushbu Pahwa, Satya Sai Srinath Namburi, Manish Shrivastava, Maneesh Singh, Bapi S. Raju, Manish Gupta |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20029
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain) Subba Reddy Oota, Akshett Jindal, Ishani Mondal, Khushbu Pahwa, Satya Sai Srinath Namburi, Manish Shrivastava, Maneesh Singh, Bapi S. Raju, Manish Gupta Abstract Transformer-based language models, though not explicitly trained to mimic brain recordings, have demonstrated surprising alignment with brain activity. Progress in these models—through increased size, instruction-tuning, and multimodality—has led to better representational alignment with neural data. Recently, a new class of instruction-tuned multimodal LLMs (MLLMs) have emerged, showing remarkable zero-shot capabilities in open-ended multimodal vision tasks. However, it is unknown whether MLLMs, when prompted with natural instructions, lead to better brain alignment and effectively capture instruction-specific representations. To address this, we first investigate brain alignment, i.e., measuring the degree of predictivity of neural visual activity using text output response embeddings from MLLMs as participants engage in watching natural scenes. Experiments with 10 different instructions (like image captioning, visual question answering, etc.) show that MLLMs exhibit significantly better brain alignment than vision-only models and perform comparably to non-instruction-tuned multimodal models like CLIP. We also find that while these MLLMs are effective at generating high-quality responses suitable to the task-specific instructions, not all instructions are relevant for brain alignment. Further, by varying instructions, we make the MLLMs encode instruction-specific visual concepts related to the input image. This analysis shows that MLLMs effectively capture count-related and recognition-related concepts, demonstrating strong alignment with brain activity. Notably, the majority of the explained variance of the brain encoding models is shared between MLLM embeddings of image captioning and other instructions. These results suggest that enhancing MLLMs’ ability to capture task-specific information could lead to better differentiation between various types of instructions, and thereby improving their precision in predicting brain responses. We make the code publicly available 1 1 1 https://github.com/subbareddy248/mllm_instruction_brain . 1 Introduction Brain encoding aims at constructing neural brain activity recordings given an input stimulus. Prior studies in brain encoding have demonstrated that representations from multimodal models, which align multiple modalities (e.g., vision and language), achieve a higher degree of brain alignment for both image-based and text-based representations compared to vision-only models, particularly when using naturalistic image stimuli ( Doerig et al. 2022 ; Wang et al. 2022 ; Oota et al. 2022b ; Popham et al. 2021 ; Oota et al. 2024b ) . Specifically, these studies have shown that representations from multimodal models are better at predicting neural responses in the high-level visual cortex as compared to previous vision-only models like convolutional neural networks (CNNs) ( Wang et al. 2022 ; Popham et al. 2021 ) . However, prior research investigating the effectiveness of multimodal models for brain alignment has primarily relied on image-caption pairs, leaving the full potential of multimodal models, especially when enhanced with large language models (LLM) and task-specific instructions, underexplored. In this paper, we explore how effectively multimodal representations, obtained by prompting MLLMs with various natural instructions, align with visual processing brain regions. Several previous unimodal studies have found that Transformer models finetuned for specific tasks more closely align with brain processes during language comprehension ( Oota et al. 2022a ; Aw & Toneva 2023 ; Sun & Moens 2023 ; Oota et al. 2024c ) , speech processing ( Oota et al. 2023 ; Tuckute et al. 2023 ; Oota et al. 2024a ) , and visual processing ( Wang et al. 2019 ; Conwell et al. 2022 ) , yielding better brain predictivity than pretrained models. However, these studies rely on separate task-specific models, which limits generalization, as each task-specific representations are aligned with the same human brain recordings. Furthermore, finetuning data needs to be obtained for each task, and a new model needs to be trained separately. Recently, instruction-tuning has become a widely adopted method for fine-tuning the same baseline large language model across multiple different natural language processing (NLP) tasks. This approach has been shown to outperform task-specific models ( Taori et al. 2023 ; Touvron et al. 2023 ; Jiang et al. 2023 ; Abdin et al. 2024 ; Dubey et al. 2024 ) , and demonstrates improved brain alignment compared to smaller language models ( Sun et al. 2023 ; Sun & Moens 2023 ; Loong Aw et al. 2024 ) . Building on these advances, researchers have extended instruction-tuning to multimodal LLMs (MLLMs) ( Xu et al. 2023 ; Dai et al. 2023 ; Liu et al. 2024 ) , enabling impressive unimodal and multimodal capabilities. These progressive improvements motivate us to explore the effectiveness of instruction-tuned MLLMs for brain encoding. By leveraging task-specific representations from a single MLLM, we aim to capture multiple aspects of an image beyond simple captioning, such as people, foreground and background elements, interactions between objects, environments, colors, food items, animals, and outdoor scenes. This leads to a critical question related to understanding of human-alignment of AI: Do these multimodal instruction-tuned models prompted using natural language instructions lead to better brain alignment and differentiate instruction-specific representations? To address this, we investigate different ways of prompting MLLMs with various task-specific instructions. Overall, this research utilizes various task-specific MLLM representations to develop encoding models based on fMRI responses within a multimodal model framework (see Fig. 1 for workflow). Figure 1: Leveraging instruction-tuned multimodal LLMs for brain encoding with a diverse set of instructions. For the given image, we could obtain different multimodal representations using instructions that ask the model to (i) generate the caption of the image, (ii) identify whether people are present, or (iii) determine the primary colors dominant in the image. Using instruction-specific representations, we estimate the alignment using a simple linear function f (ridge regression) which map MLLM representations to brain recordings. Using brain recordings of participants watching natural scenes images from the NSD dataset ( Allen et al. 2022 ) , we investigate several research questions. First, we explore the effectiveness of MLLM representations and compare their brain alignment with unimodal and multimodal models. For the purposes of this work, we focus on three MLLMs (InstructBLIP ( Dai et al. 2023 ) , mPLUG-Owl ( Ye et al. 2023 ) and IDEFICS ( Laurençon et al. 2023 ) ), one image-based (ViT-H) and one multimodal model (CLIP). We probe these MLLMs using ten different instructions across six visual tasks. Specifically, we investigate which of these task-specific instructions result in better brain alignment. Second, do the instruction-specific representations from MLLMs differentiate the visual regions that process this information, thereby aligning with the mechanisms of human visual cognition? Third, do task-specific instructions from MLLMs account for visual concepts understanding, and which brain regions are responsible for processing different visual concepts? Fourth, we use a variance partitioning approach to assess the unique and shared variance of each task-specific instruction to brain responses. This analysis provides insights into how different visual tasks complement or overlap in explaining brain activity, thereby enhancing our understanding of the functional organization of visual processing in the brain. Our analysis of instruction-tuned MLLMs and brain alignment reveals several key conclusions: (i) MLLMs demonstrate significantly better brain alignment than vision-only models and perform comparably to multimodal models such as CLIP. (ii) While all three MLLMs generate task-specific output tokens based on instructions, not all instructions contribute to higher brain alignment. Specifically, the