Letting the neural code speak: Automated characterization of monkey visual neurons through human language

Vedang Lad 1 &Katrin Franke 1,2 &Tamar Rott Shaham 3 &Surya Ganguli 1 and Andreas S. Tolias 1 &Sophia Sanborn 1 &Nikos Karantzas 1 and 1 Stanford University 2 University of Tübingen 3 Massachusetts Institute of Technology · 2026

Read Letting the neural code speak: Automated characterization of monkey visual neurons through human language by Vedang Lad 1 &Katrin Franke 1,2 &Tamar Rott Shaham 3 &Surya Ganguli 1 and Andreas S. Tolias 1 &Sophia Sanborn 1 &Nikos Karantzas 1 and 1 Stanford University 2 University of Tübingen 3 Massachusetts Institute of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12485.txt.

Introduction

Lad and six colleagues use plain language to describe what single monkey visual neurons encode. In primary visual cortex, maths models already capture selectivity, but higher areas lacked a matching framework. Using digital twins of areas V1 and V4, they run a closed loop: captions of high- and low-activating images feed a semantic hypothesis, which is tested with synthesised images. Most neurons gain concise, verifiable descriptions, from edges and spatial frequency in V1 to form, colour and texture mixes in V4, and generated images drive V4 neurons as predicted.

Catalog record

CREATORVedang Lad 1 &Katrin Franke 1,2 &Tamar Rott Shaham 3 &Surya Ganguli 1 and Andreas S. Tolias 1 &Sophia Sanborn 1 &Nikos Karantzas 1 and 1 Stanford University 2 University of Tübingen 3 Massachusetts Institute of Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89130
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12485

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Letting the neural code speak: Automated characterization of monkey visual neurons through human language Vedang Lad 1 &Katrin Franke 1,2 &Tamar Rott Shaham 3 &Surya Ganguli 1 and Andreas S. Tolias 1 &Sophia Sanborn 1 &Nikos Karantzas 1 and 1 Stanford University 2 University of Tübingen 3 Massachusetts Institute of Technology Abstract Understanding what individual neurons encode is a core question in neuroscience. In primary visual cortex (V1), mathematical models (e.g., Gabor functions) capture neural selectivity, but no comparable framework exists for higher areas. We show that natural language can fill this role: across macaque V1 and V4, the selectivity of most neurons is captured by concise, verifiable semantic descriptions. Using digital twins of V1 and V4, we develop a closed-loop framework that translates each neuron’s high- and low-activating images into dense captions, generates a semantic hypothesis and synthesized images, and verifies the hypothesis in silico. Descriptions range from oriented edges and spatial frequency in V1 to conjunctions of form, color, and texture in V4. In V4, images generated from activating and suppressing hypotheses drove 96.1% of neurons above the 95th and 97.6% below the 5th percentile of natural-image responses, respectively (vs. ∼ \sim 10% for random images); V1 activation results matched V4, while V1 suppression was less describable in language. Representational similarity analysis reveals partial alignment between neural activity, vision embeddings, and language embeddings, with vision most aligned to neural activity; alignment lost in the text bottleneck is recovered when hypotheses are rendered back into images, showing that linguistic compression is lossy yet semantically faithful. Together, these results show that combining generative models with neural digital twins enables interpretable, testable descriptions of neural function at scale, toward agentic scientific discovery. Code Website 1 Introduction A central goal of neuroscience is to understand what information individual neurons represent and how their response properties give rise to perception. In early visual cortex, this question has tractable answers: neurons in primary visual cortex (V1) are well described by Gabor functions ( 31 ; 15 ) , capturing orientation and spatial frequency tuning with clear connections to efficient coding principles ( 53 ; 69 ) . Beyond V1, no comparable general framework exists. Neurons in higher visual areas like V4 and inferotemporal cortex exhibit selectivity for curvature ( 55 , e.g.) , texture ( 20 , e.g.) , faces ( 77 , e.g.) , and feature conjunctions ( 9 , e.g.) , but these characterizations have emerged individually from curated stimulus sets designed to test specific hypotheses. Functional “digital twin” models have transformed how we probe neural selectivity. Deep networks trained on single-neuron responses to natural stimuli predict activity with high accuracy ( 78 ; 85 ; 11 ; 46 ) , and when combined with modern recording technologies ( 70 ; 16 ) , they enable in-silico screening of millions of stimuli across hundreds or thousands of neurons ( 19 ) . However, prediction does not yield interpretation: identifying images that drive a neuron does not reveal what visual features are encoded by the neuron. Therefore, despite being the most accurate mathematical models of neuronal tuning now available, these deep networks remain opaque. This raises a central challenge: whether neural selectivity, as captured by such models, can be distilled into a compact human-interpretable description. This gap between predictive accuracy and mechanistic understanding is not unique to neuroscience. Across scientific domains where large-scale measurement has outpaced mechanistic interpretation, such as protein structure prediction ( 32 ) or weather forecasting ( 6 ) , models achieve remarkable predictive performance while leaving the underlying principles opaque. This reflects broader tension between forms of description: simple mathematical models are interpretable but limited in scope, whereas highly expressive models capture complex phenomena but resist human understanding. Figure 1: Framework for translating neural selectivity into interpretable semantic hypotheses. The pipeline consists of three stages: Translate: Each image is converted into a detailed textual description using Gemini 3.0 Pro. To evaluate the fidelity of this image-to-text translation, we regenerate images from the captions using a text-to-image model and quantify correspondence to the original image in an image-similarity space. Semantic Hypothesis: For each neuron in our macaque V1 and V4 datasets ( 19 ) , we use a functional digital twin to screen a large-scale naturalistic image database (1.2 million ImageNet images) and identify the stimuli predicted to most strongly drive or suppress it. Gemini 3.0 Pro then analyzes the captions of these extreme response sets to derive a concise, human-interpretable semantic hypothesis describing the neuron’s selectivity. Verification: We convert the derived hypothesis into prompts for a text-to-image model, generate new images, and use the digital twin to predict neural responses. Agreement between predicted and observed activity provides a closed-loop validation of the neuron’s hypothesized semantic tuning. Here, we ask whether large language models can bridge this gap, by translating the behavior of complex predictive models into human-interpretable descriptions. In visual cortex, especially at mid to high levels, the features that define neural selectivity (shapes, textures, object parts, and their conjunctions) are features that humans routinely describe in words. We test this idea at two levels. First, at the level of individual neurons, we examine whether language can capture the features that drive and suppress neural activity strongly enough that images regenerated from the description elicit comparable responses to each neuron’s preferred and non-preferred natural images. Second, at the population level, we test whether language preserves the relational structure of the neural population code, such that similarity in neural response space is reflected in language embedding space. If so, language would constitute a principled coordinate system for neural selectivity rather than a mere labeling scheme. We test this directly in macaque V1 (n = 438 neurons) and V4 (n = 205 neurons). For each neuron, we screen 1.2 million ImageNet images through a digital twin to isolate the 15 most- and 15 least-activating stimuli, and use Gemini 3.0 Pro to translate each set into a concise natural-language hypothesis. We then verify each hypothesis in a closed loop against the digital twin: Gemini 3.0 Pro generates prompts describing the visual features that drive or suppress each neuron, Imagen 4.0 synthesizes novel images based on these prompts, and the digital twin measures their predicted responses, with affine transformations to align each stimulus to the preferred position, scale, and orientation of each neuron’s receptive field. The recovered hypotheses span the visual hierarchy, from oriented edges and spatial frequency tuning in V1 to conjunctions of form, color, and texture in V4, and hypothesis-generated images drove > 96% of neurons in both V1 and V4 above the 95th percentile of natural image responses; by contrast, randomly drawn ImageNet images reached this threshold for only 10% of neurons. Suppressive hypotheses drove 97.6% of V4 neurons below the 5th percentile, but were substantially weaker in V1 (56%), maybe reflecting the limits of linguistic expressibility for sub-lexical features. Using representational similarity analysis over neural activity, DINOv3 visual embeddings ( 68 ) , and Qwen 0.6B language embeddings ( 89 ) , we further find a partially shared geometric organization across modalities: language not only labels individual neurons but reflects the structure of the neural code. 2 Related Work Automated interpretability in machine learning The field of vision model

Read the complete text