NEAT: Concept driven Neuron Attribution in LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Natural language processing
Vivek Hruday Kavuri, Gargi Shroff, Rahul Mishra · 2025
Read NEAT: Concept driven Neuron Attribution in LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Natural language processing by Vivek Hruday Kavuri, Gargi Shroff, Rahul Mishra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15875.txt.
Introduction
Vivek Hruday Kavuri, Gargi Shroff, and Rahul Mishra present a 2025 paper on finding the neurons behind ideas in large language models. Knowing which neurons drive a prediction helps open the black box. Past neuron hunts could not capture whole concepts, and they cost much compute. The new method, NEAT, uses concept vectors to tag the neurons that stand for a given concept.
The cost drops sharply: passes through the model fall from a product of neurons and examples to just the neuron count. Tests against rival methods showed better results at lower cost. Clustering tricks sped the search further. The team also shut off found neurons to study hate speech and bias, with an Indian context check. Such switches may guide later fixes.
Catalog record
| CREATOR | Vivek Hruday Kavuri, Gargi Shroff, Rahul Mishra |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15875
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NEAT: Concept driven Neuron Attribution in LLMs DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2025; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Natural language processing Vivek Hruday Kavuri, Gargi Shroff, Rahul Mishra Abstract . Locating neurons that are responsible for final predictions is important for opening the black-box large language models and understanding the inside mechanisms. Previous studies have tried to find mechanisms that operate at the neuron level but these methods fail to represent a concept and there is also scope for further optimization of compute required. In this paper, with the help of concept vectors, we propose a method for locating significant neurons that are responsible for representing certain concepts and term those neurons as concept neurons. If the number of neurons is n and the number of examples is m, we reduce the number of forward passes required from O(n*m) to just O(n) compared to the previous works and hence optimizing the time and computation required over previous works. We also compare our method with several baselines and previous methods and our results demonstrate better performance than most of the methods and are more optimal when compared to the state-of-the-art method. We, as part of our ablation studies, also try to optimize the search for the concept neurons by involving clustering methods. Finally, we apply our methods to find, turn off the neurons that we find, and analyze its implications in parts of hate speech and bias in LLMs, and we also evaluate our bias part in terms of Indian context. Our methodology, analysis and explanations facilitate understating of neuron-level responsibility for more broader and human-like concepts and also lay a path for future research in this direction of finding concept neurons and intervening them. 1. Introduction Large Language Models have demonstrated remarkable capabilities across a wide range of natural language processing tasks. However, the internal mechanisms by which these models represent and utilize abstract concepts remain largely opaque. Understanding which internal components are responsible for specific behaviors is a central goal of mechanistic interpretability , as it can enhance transparency, improve controllability, and enable targeted interventions in model behavior. Figure 1. Overall architecture of NEAT: from obtaining the concept vector to identifying and ablating concept-relevant neurons. A key challenge in this space is neuron attribution — identifying individual neurons or small sets of neurons that significantly influence the model’s predictions. While many attribution methods have been proposed, including causal tracing ( Vig et al. 2020 ; Meng et al. 2022 ; Zhang and Nanda 2024 ) and gradient-based approaches ( Sundararajan et al. 2017 ; Lundstrom et al. 2022 ) , these techniques often require multiple forward and/or backward passes per example, making them computationally expensive at the neuron level. As a result, most prior work has focused on attribution at the hidden state level rather than on fine-grained neuron analysis. Recent advances, such as static neuron importance scoring ( Yu and Ananiadou 2024a ) , reduce the computational burden by avoiding multi-pass gradient calculations. While effective for single-instance analysis, these methods do not explicitly address the identification of neurons that encode abstract, human-interpretable concepts across diverse contexts. This limitation is particularly important for applications such as bias detection, harmful content mitigation, and concept-level interpretability, where generalization beyond a single example is essential. In this work, we address this gap by introducing a concept-vector-based approach for locating neurons that store information about specific concepts. Our method significantly reduces the computational cost compared to prior works — lowering the forward pass complexity from O(n\times m) to O(n) , where n is the number of neurons and m is the number of examples. This is achieved by representing a concept as a single dense vector in the model’s semantic space, then probing all neurons in a single pass to identify those most strongly associated with the concept. We term such neurons concept neurons . Beyond identification, we investigate clustering as an ablation strategy to uncover whether concept-related neurons are organized into semantically coherent groups. This allows us to explore group-level interventions, which can be more efficient and interpretable than modifying neurons individually. We apply our methodology to two real-world settings: hate speech and gender bias , with the latter including a case study in the Indian socio-cultural context. Our contributions are as follows: • We propose a novel, computationally efficient method for identifying neurons that represent human-interpretable concepts, reducing forward pass complexity from O(n\times m) to O(n) . • We introduce a clustering-based ablation strategy to study group-level neuron interventions. • We apply our approach to analyze and intervene on hate speech and gender bias in LLMs, providing both quantitative and qualitative evaluations, including a domain-specific bias analysis for the Indian context. Our results demonstrate that the proposed method not only outperforms several baselines in identifying concept-relevant neurons but also provides an interpretable and efficient framework for neuron-level analysis and intervention. 2. Related Work 2.1. Attribution Methods for Transformers Identifying the internal components of transformer models that are most influential for a given prediction is a longstanding research challenge. Causal tracing approaches ( Pearl 2022 ; Vig et al. 2020 ; Meng et al. 2022 ; Goldowsky-Dill et al. 2023 ; Zhang and Nanda 2024 ; Wu et al. 2023 ) and Gradient-based attribution methods ( Sundararajan et al. 2017 ; Kindermans et al. 2022 ; Miglani et al. 2020 ; Lundstrom et al. 2022 ) are widely used to quantify the influence of internal modules in final predictions, but these techniques often require multiple forward and/or backward passes per input making them computationally expensive, especially when applied at the neuron level. Consequently, prior work often focuses on hidden state attribution ( Meng et al. 2022 ; Geva et al. 2023 ; Stolfo et al. 2023 ) rather than fine-grained neuron analysis. Recent work by ( Yu and Ananiadou 2024a ) proposes a static method for identifying important neurons in LLMs, designed to operate without the heavy computational cost of multi-pass gradient or causal tracing techniques by defining an importance score based on the log-probability increase when a neuron’s activation is added to the layer output. While this method is effective for analyzing neuron importance in a single input instance, it doesn’t directly address the identification of neurons that store knowledge about broader, abstract concepts across diverse inputs which is crucial for tasks like bias detection and concept-level interpretability, where generalization beyond a single example is required. Another type of studies computes attribution in a single forward pass, relying on proxy saliency measures such as attention weights ( Vig et al. 2020 ; Jaunet et al. 2022 ; Yeh et al. 2024 ; Wang et al. 2023 ; Liu et al. 2024 ) or feed-forward neuron’s coefficient scores. However, these approaches face persistent concerns regarding faithfulness and interpretability ( Geva et al. 2022 ; Lee et al. 2024 ) , due to lack of standardized evaluation frameworks. 2.2. Mechanistic Interpretability Mechanistic interpretability aims to reverse-engineer the internal computations of language models, often by projecting intermediate activations into the vocabulary space to reveal interpretable patterns ( Geva et al. 2021 ; Dar et al. 2023 ; Pal et