On the Analogy between Human Brain and LLMs: Spotting Key Neurons in Grammar Perception
Sanaz Saki Norouzi, Mohammad Masjedi, Pascal Hitzler · 2025
Read On the Analogy between Human Brain and LLMs: Spotting Key Neurons in Grammar Perception by Sanaz Saki Norouzi, Mohammad Masjedi, Pascal Hitzler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06519.txt.
Introduction
Sanaz Saki Norouzi, Mohammad Masjedi, and Pascal Hitzler find grammar-specialized neurons inside a large language model, echoing a known brain pattern. Neuroscience shows different word classes engage different neurons in human language areas. Whether artificial networks, loosely inspired by brains, sort grammar the same way was an open question.
Working with the Llama 3 model, the team pinpointed the neurons most tied to predicting words of each part of speech. A classifier trained on just those neurons' activity predicted word classes on fresh text reliably. The results point to a dedicated grammar subspace inside the model, reminiscent of patterns seen in human lesion studies.
Catalog record
| CREATOR | Sanaz Saki Norouzi, Mohammad Masjedi, Pascal Hitzler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06519
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Analogy between Human Brain and LLMs: Spotting Key Neurons in Grammar Perception Sanaz Saki Norouzi, Mohammad Masjedi, Pascal Hitzler Abstract Artificial Neural Networks, the building blocks of AI, were inspired by the human brain’s network of neurons. Over the years, these networks have evolved to replicate the complex capabilities of the brain, allowing them to handle tasks such as image and language processing. In the realm of Large Language Models, there has been a keen interest in making the language learning process more akin to that of humans. While neuroscientific research has shown that different grammatical categories are processed by different neurons in the brain, we show that LLMs operate in a similar way. Utilizing Llama 3, we identify the most important neurons associated with the prediction of words belonging to different part-of-speech tags. Using the achieved knowledge, we train a classifier on a dataset, which shows that the activation patterns of these key neurons can reliably predict part-of-speech tags on fresh data. The results suggest the presence of a subspace in LLMs focused on capturing part-of-speech tag concepts, resembling patterns observed in lesion studies of the brain in neuroscience. Introduction As Large Language Models (LLMs) continue to show significant performance on various NLP tasks, understanding how they behave internally has gained attention in AI research. Researchers try to understand how LLMs store specific linguistic or factual knowledge, and how this knowledge can be interpreted or even controlled ( 35 ; 41 ) . To support these efforts, Explainable AI (XAI) techniques are employed to help researchers understand what internal components cause specific outputs and why certain predictions are made ( 43 ) . To this end, different explanation methods are explored such as gradient-based ( 7 ) , attention-based ( 21 ) , and neuron-activation-based ( 1 ) . However, it is proven that attention analysis cannot be interpreted as explanations ( 20 ) . In contrast, gradient-based approaches have demonstrated greater promise in identifying important neurons ( 3 ) . On the other hand, activation analysis methods have some limitations as raw activation values may not reliably indicate neuron importance, and there are no ground-truth explanations for the neurons based on activation ( 43 ) . Recent studies have primarily explored how different languages are localized inside multilingual LLMs and tried to find important neurons linked to each language or domain ( 42 ; 3 ) . However, the identification of specific regions related to fine-grained neuron-level grammar encoding has largely remained unexplored, especially in decoder-only models such as Llama. In parallel, there have been studies in cognitive neuroscience to illustrate the functions of the human brain. Earlier, the researchers believed in a modular view of the brain, in which specific brain regions are associated with specific functions ( 30 ; 17 ) . More recent analyses provided insights into connectionism, which defines brain functions as the result of interactions between regions rather than isolated parts. In other words, a task like language understanding involves multiple regions and not just one module ( 33 ; 2 ) . However, recent studies have emphasized the integrative view which combines modularity and connectionism views. In these studies, researchers have shown that cognitive functions are not confined to a single region, but there are interactions between neural units within a specific region and across the brain ( 34 ; 11 ) . Artificial neural networks are inspired by the human brain, and Large Language Models represent an advanced and complex form of these networks. LLMs are now utilized across a wide range of tasks and have demonstrated significant impact. Thus, understanding how they work can help us explore similarities between them and the human brain. In this paper, we provide the model with incomplete sentences and analyze its next-word predictions along with their corresponding part-of-speech (POS) tags. Using XAI approaches, we will highlight that there is an analogy between the human brain and the LLMs in terms of grammar perception. First, combining gradient-based attribution scores with statistical significance, we identify neurons that are significantly sensitive to these grammatical concepts. Next, using machine learning analyses, we show that these neurons consistently represent interpretable grammatical concepts. Overall, the key contributions of this study are: • Neural grammatical mapping in LLMs: We demonstrate neuron-level specialization for POS tags in Llama-3. • Concept-based interpretation: Using concept-based analysis and in-depth statistical methods, we provide evidence that the identified neurons encode POS tag concepts • Dataset construction: In addition to the main contributions, a suitable dataset for the POS tag concept analysis is built. Taken together, our study shows that neurons associated with grammatical categories are distributed across all layers of the LLM, highlighting the connectionist perspective of the human brain. But more interestingly, it also indicates that there are certain neurons within each layer that are highly specialized in the perception of grammatical concepts, aligning with the latest discoveries in cognitive neuroscience about the functionality of the brain. Related Work Knowledge Representation in the Human Brain Understanding how the human brain organizes and localizes linguistic knowledge has been a key question in cognitive neuroscience. Researchers have employed techniques such as brain lesion studies and neuroimaging to study how different language functions such as grammar, syntax, and meaning are handled by specific or partially distributed areas of the brain ( 15 ) . For example, 38 , investigated how the brain processes different grammatical categories, focusing on nouns and verbs. They discovered that the brain processes nouns and verbs differently, not only based on their meaning, but also their grammatical roles. And while some general language areas are shared, these word categories also activate distinct neural regions ( 38 ) . Another study introduced concept cells as selective neurons that encode abstract and invariant representations of specific concepts, ( 37 ) . In ( 19 ) , it was shown that during natural narrative listening, the brain encodes rich semantic content in spatially and semantically organized cortical maps. 9 found that there is a shared neural basis for language meaning across modalities ( 9 ) . Another research showed that single brain neurons in humans respond to meaningful concepts in visual and textual format ( 36 ) . Other studies have proposed that human conceptual knowledge is shaped by the integration of sense experiences and abstract language-based information, supported by distinct but interacting neural systems ( 40 ; 27 ; 24 ) . Knowledge Representation in LLMs LLMs have demonstrated they can handle a wide range of tasks well, which shows that they understand the structure and meaning of language ( 31 ) . Thus, researchers are trying to figure out how these models store and use knowledge. An area of investigation has focused on evaluating general factual knowledge encoded in LLMs ( 35 ; 23 ; 29 ) , or domain-specific knowledge such as mathematical capabilities ( 10 ) . In the linguistic field, ( 28 ; 22 ) , analyzed BERT’s layer-level representations using probing tasks to investigate the extent to which they encode syntactic and structural properties of language. Other studies have investigated identifying where and how subject-verb agreement is represented in multilingual and monolingual BERT models using causal mediation analysis (