Neuron Empirical Gradient: Discovering and Quantifying Neurons’ Global Linear Controllability

Xin Zhao, Zehui Jiang, Naoki Yoshinaga · 2024

Read Neuron Empirical Gradient: Discovering and Quantifying Neurons’ Global Linear Controllability by Xin Zhao, Zehui Jiang, Naoki Yoshinaga (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18053.txt.

Introduction

Xin Zhao, Zehui Jiang and Naoki Yoshinaga study how single neurons in pre-trained language models shape model output, beyond the few strong neurons past work chased. Using interventions on a knowledge test set, they find a broad linear link between neuron activations and outputs across the network. The slope of that link is what they name the neuron empirical gradient.

Their tool, NeurGrad, computes that gradient at large scale. On a multi-genre knowledge benchmark of eight thousand questions, the gradient captured model knowledge well. Probes built on it also tracked language skills across prompts. The team notes four traits of the representation: it is efficient, robust, flexible, and its parts depend on each other.

Catalog record

CREATORXin Zhao, Zehui Jiang, Naoki Yoshinaga
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18053

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neuron Empirical Gradient: Discovering and Quantifying Neurons’ Global Linear Controllability Xin Zhao, Zehui Jiang, Naoki Yoshinaga Abstract While feed-forward neurons in pre-trained language models (PLMs) can encode knowledge, past research targeted a small subset of neurons that heavily influence outputs. This leaves the broader role of neuron activations unclear, limiting progress in areas like knowledge editing. We uncover a global linear relationship between neuron activations and outputs using neuron interventions on a knowledge probing dataset. The gradient of this linear relationship, which we call the neuron empirical gradient (NEG) , captures how changes in activations affect predictions. To compute NEG efficiently, we propose NeurGrad , enabling large-scale analysis of neuron behavior in PLMs. We also show that NEG effectively captures language skills across diverse prompts through skill neuron probing. Experiments on MCEval8k , a multi-genre multiple-choice knowledge benchmark, support NEG’s ability to represent model knowledge. Further analysis highlights the key properties of NEG-based skill representation: efficiency, robustness, flexibility, and interdependency. The code and data are released. xzhao-tkl/NEG iszhaoxin/MCEval8K 1 Introduction Pre-trained language models (PLMs) based on Transformer architecture Vaswani et al. (2017) effectively encode human knowledge, prompting efforts to understand their inner workings. While previous studies have shown that the feed-forward (FF) neurons play key roles in encoding factual knowledge ( Dai et al., 2022 ; Yu and Ananiadou, 2024 ) and general language skills ( Wang et al., 2022 ; Tan et al., 2024 ) , they face two main challenges. First, current methods mostly rank neurons by importance without measuring the link between neuron activations and model outputs ( Dai et al., 2022 ; Meng et al., 2022 ; Yu and Ananiadou, 2024 ) , limiting use cases like knowledge editing Zhang et al. (2024) . Second, these methods are costly, involving repeated activation modifications ( Dai et al., 2022 ; Meng et al., 2022 ; Goldowsky-Dill et al., 2023 ) or extensive tensor operations ( Yu and Ananiadou, 2024 ) , making them inefficient for analyzing all neurons on large models. Figure 1: Our contributions: i) revealing linearity between activation and output shifts, ii) proposing NeurGrad, an efficient method to quantify it, iii) confirming that NEGs capture language skills on MCEval8K. This study begins by quantitatively analyzing how neuron activations influence model outputs (RQ1) (Figrue 1 ). Using MyriadLAMA ( Zhao et al., 2024 ) , a factual knowledge probing dataset, on seven PLMs, including large language models (LLMs) like Llama2-70B (§ 2 ), we gradually modify random neuron activations and observe the changes in the token probabilities for correct knowledge (hereafter, output shifts ). Notably, we find that within a certain range, shifts in neuron activations (hereafter, activation shifts ) have a linear impact on output shifts. We define and quantify the gradient of this linear relationship as neuron empirical gradient (NEG) , enabling quantitative neuron analysis. Next, we explore whether shifting neuron activations can precisely control PLMs’ output probabilities (RQ2) . Since computing NEGs requires costly inference, we propose NeurGrad (§ 3 ), an efficient method for estimating a single neuron’s NEG. It builds on the empirical finding that computational gradients (§ 3.1 ) strongly correlate with NEG magnitudes, though less so with directions. Using MyriadLAMA, we validate NeurGrad’s performance against ground-truth NEGs, showing it outperforms existing neuron-ranking methods ( Dai et al., 2022 ; Yu and Ananiadou, 2024 ) . With NeurGrad, we further examine multi-neuron control, finding that NEGs can accumulate across neurons (§ 4 ), while the effect weakens as more neurons are involved or larger activation shifts are involved. Finally, we examine whether NEGs can capture general language skills beyond factual knowledge (RQ3) . To explore this, we perform skill neuron probing ( Wang et al., 2022 ; Song et al., 2024 ) using NEGs. As the previous studies on skill neuron probing focus on limited language skills, we introduce MCEval8K , a benchmark covering six genres and 22 tasks for broad LLM evaluation. Our results show that NEGs can represent a wide range of language skills. Furthermore, our in-depth analysis highlights the properties of NEG-based skill neurons, including the efficiency in representing language skills, robustness when facing diverse contexts, substitutability of skills being represented by different neurons, and interdependency between neurons in representing skills. Our contributions (Figrue 1 ) are as follows: • We confirm that activation and output shifts are linearly correlated within a certain range through neuron intervention, defined as neuron empirical gradient (NEG) (§ 2 ). • We introduce NeurGrad , an efficient method to NEG (§ 3 ), and conduct in-depth analysis about neuron controllability (§ 4 ). • We show that NEGs can represent language skills through skill neuron probing (§ 5 ) ; skill neurons exhibit efficiency, robustness, inclusivity, and interdependency (§ 6 ). • We develop MCEval8K , a multiple-choice benchmark covering six genres and 22 language understanding tasks (§ 5.2 , § F ). 2 Neuron Linearity to Model Output This section empirically answers how neurons in PLMs’ FF layers influence model outputs. We observe the resulting change in output tokens’ probabilities for fine-grained neuron-level interventions. 2.1 Neuron-level Intervention Experiments Models. To ensure the generality of our findings, we evaluate both masked and causal LMs with varying sizes and learning strategies. For masked LMs, we use two BERT Devlin et al. (2019) models, BERT base and BERT large , and have them predict masked tokens. For causal LMs, we examine five LLMs with different model sizes and language families, including Llama2 ( Touvron et al., 2023 ) (7B, 70B), Llama3.1 (8B), Llama3.2 (3B) ( Grattafiori et al., 2024 ) , and Qwen2.5 (7B) ( Qwen et al., 2025 ) . All of these models are instruction-tuned. Following Zhao et al. (2024) , we use a zero-shot prompt to generate single-token answers. See § E for details. Dataset. We use MyriadLAMA ( Zhao et al., 2024 ) , a multi-prompt knowledge probing dataset for neuron intervention. Its diverse prompts help reduce bias from specific phrasing. We focus on single-token probing, where the target answer is a single token. For each PLM, we randomly sample 1000 prompts from MyriadLAMA, where the model predicts the token representing the correct answer. Due to tokenizer and knowledge differences, the sampled prompts vary across PLMs. Neuron intervention. We shift activations in the range of [-10, 10] 1 1 1 The range mostly covers the distribution of activations; see § B.1 for details. with the step size of 0.2 to track changes in target token probabilities. For example, consider the following prompt: Predict the [MASK] in each sentence in one word. Q: [MASK] is the capital of Japan. A: We modify the activations of specific neurons and observe how the probability of the target word Tokyo changes at the final position. Since conducting one intervention experiment for a neuron-token-prompt combination requires 100 inferences due to the shift values, we randomly sample 2 2 2 The actual number of randomly sampled neurons will be given in each experiment. neurons to make computation tractable while ensuring broad coverage. Figure 2: Average absolute correlation between activation and output shifts. Results. To understand how output shifts respond to neuron activation shifts, we compute the Pearson correlation coefficient ( r ) between activation shifts

Read the complete text