Transformer brain encoders explain human high-level visual responses

Hossein Adeli, Minni Sun, Nikolaus Kriegeskorte · 2025

Read Transformer brain encoders explain human high-level visual responses by Hossein Adeli, Minni Sun, Nikolaus Kriegeskorte (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17329.txt.

Introduction

Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte modeled the visual brain with focus in a 2025 paper. Standard brain coding fits a huge linear map from a neural network's traits to each brain site. Ignoring the layout of traits in both systems, while factored versions assume fixed receptive fields that suit only early vision. Their model brain encoder instead routes retinotopic traits dynamically, using focus to send the right image information to category-selective areas downstream.

Across feature backbones and recording modalities, the focus model foretold brain activity during natural scene viewing by a lot better than rivals. Better still, its routing signals can be visualized per category area for any image, making the model's account of high-level vision straight inspectable.

Catalog record

CREATORHossein Adeli, Minni Sun, Nikolaus Kriegeskorte
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17329

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transformer brain encoders explain human high-level visual responses Hossein Adeli, Minni Sun, Nikolaus Kriegeskorte Abstract A major goal of neuroscience is to understand brain computations during visual processing in naturalistic settings. A dominant approach is to use image-computable deep neural networks trained with different task objectives as a basis for linear encoding models. However, in addition to requiring tuning a large number of parameters, the linear encoding approach ignores the structure of the feature maps both in the brain and the models. Recently proposed alternatives have focused on decomposing the linear mapping to spatial and feature components but focus on finding static receptive fields for units that are applicable only in early visual areas. In this work, we employ the attention mechanism used in the transformer architecture to study how retinotopic visual features can be dynamically routed to category-selective areas in high-level visual processing. We show that this computational motif is significantly more powerful than alternative methods in predicting brain activity during natural scene viewing, across different feature basis models and modalities. We also show that this approach is inherently more interpretable, without the need to create importance maps, by interpreting the attention routing signal for different high-level categorical areas. Our approach proposes a mechanistic model of how visual information from retinotopic maps can be routed based on the relevance of the input content to different category-selective regions. Code available at github.com/hosseinadeli/transformer brain encoder . 1 Introduction An influential approach to study plausible neural computations in the brain is to train Deep Neural Network (DNN) models on different tasks [ 34 , 21 ] and compare their learned representation to brain activity [ 42 , 17 ] . There has been a great deal of discussion and research on best ways to compare the learned representations to the ones recorded from the brain (across models and across models and brains). One main approach is to build encoding models— learn a mapping function from one feature domain to another and measure the accuracy of the prediction in held-out sets [ 10 , 28 ] . An alternative approach is to characterize the geometry or topology of the representation in each model or in the brain and then compare them (e.g. RSA; [ 22 ] ). In this work, we focus on the learned encoding functions, as we believe that it can give us further insight into the computations in the brain. The visual system uses structured retinotopic maps as it processes visual information in the cortex. Not surprisingly, models, such as Convolutional and transformer neural networks, that also maintain retinotopic maps of the space perform best on different visual tasks (e.g. recognition and segmentation) and consistently outperform other models in different brain activation prediction benchmarks [ 36 , 13 ] . However the retinotopic feature maps from deep networks presents typically have a very large number of units posing us with a challenge when mapped unto the responses in the brain. Linear encoding models, although theoretically the simplest choice, can become very high-dimensional in that case (the number of parameters equals the product of the number of model units and the number of units/voxels to be predicted) and require strong regularization (L2 penalty) given the size of typical fMRI datasets [ 28 ] . To address these limitations, approaches have been proposed that learn spatial receptive fields (RF) for different units or voxels in the brain data, using which the representation is first aggregated across space and then the lower dimensional representation is linearly mapped to the brain responses [ 19 , 39 , 27 ] . These models have been shown to perform on par with linear regression models despite having a fraction of the number of parameters and are also more plausible mechanisms of how information can route to different units. However, they can only capture fixed routing where input to a unit comes from a specific area in space regardless of the input content. Transformer architectures has been extremely successful in many domains, including vision [ 9 ] and language [ 41 ] . Their success can be attributed to a general and simple (therefore scalable) computational motif where information is routed based on the content. In these models, each token (be a representation of a word in a sentence or a patch in an image) queries other tokens to find how relevant they are to updating its representation. The selective nature of this mixing has motivated naming this process "attention" in Transformers [ 41 ] . Then the new representation of this token becomes the average of the representation of all tokens, weighted by their degree of relevance (i.e. attention scores). We hypothesize that the optimal way for the routing of information from the retinotopic visual maps to category selective areas is to use the same computational motif where brain areas only attend to parts of the visual maps with the content relevant to what the area is selective for (Fig. 1 ). For example if there is face in the image, it could appear anywhere, but the FFA (fuisform face area) can learn to route only the information from the patches where the face-like stimuli are and then expand this lower dimensional representation in the area. Note that this approach is in a way a generalization of the aforementioned RF based methods going from fixed receptive fields to a dynamic content-based receptive fields. 2 Related works Brain encoding models: Predicting brain activity is an important objective, both as an engineering challenge and also as a means of studying brain computations, reflected in the number of community benchmarks such as Algonauts [ 13 ] , Brain-score [ 36 ] , and Sensorium [ 40 ] . The availability of large-scale neural datasets has necessitated innovation in new encoding models [ 16 ] . Spatial-feature decomposition models have shown that considering the retinotopic maps and the receptive field organization can lead to more efficient encoding models [ 19 , 39 , 27 , 35 ] . Generalizing these approaches to high-level visual areas would require considering more dynamic routing motifs. Self-supervised Vision Transformers: Transformers have been shown to outperform convolutional and recurrent neural networks (CNNs) on a variety of visual tasks including object recognition [ 9 ] . More recent studies have explored training these models on self-supervised objectives, yielding some intriguing object-centric properties [ 1 ] that are not as prominent in the models trained for classification. When trained with self-distillation loss (DINO, [ 4 ] and DINOv2 [ 30 ] ), the attention values contain explicit information about the semantic segmentation of the foreground objects and their parts, reflecting that these models can capture object-centric representations without labels. These findings show that features from these models can be a good basis for predicting neural activity in the brain. Recent work has also shown that networks trained using self-supervised contrastive losses (such as SimCLR; [ 7 ] ) match the predictive power of supervised models for high-level ventral-stream visual representations in the brain [ 20 , 6 ] . These works argue for self-supervised learning methods as a more plausible objective function for learning brain like visual representations. Encoder-decoder Vision Transformers: Transformer-based encoder-decoder models provide a general

Read the complete text