Emergent Specialization: Rare Token Neurons in Language Models

Jing LiuENS, Université PSL, EHESS, CNRSParis, France, Haozheng Wang, Yueheng Li · 2025

Read Emergent Specialization: Rare Token Neurons in Language Models by Jing LiuENS, Université PSL, EHESS, CNRSParis, France, Haozheng Wang, Yueheng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12822.txt.

Introduction

Jing Liu, Haozheng Wang, and Yueheng Li hunt for special neurons inside language models, in a 2025 paper. Rare words matter in expert fields, yet models handle them poorly. The team finds that a small set of neurons carries an outsized effect on how a model predicts these rare tokens.

These neurons form in three stages during training, shifting from a flat start to a split-off group with its own pattern. In the model's activity space, they switch on together and stay apart from other neurons. Their rise seems tied to the heavy-tailed spread of the model's weights. The study offers a concrete picture of how models set aside capacity for the uncommon words they still must get right.

Catalog record

CREATORJing LiuENS, Université PSL, EHESS, CNRSParis, France, Haozheng Wang, Yueheng Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12822

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Emergent Specialization: Rare Token Neurons in Language Models Jing LiuENS, Université PSL, EHESS, CNRSParis, France, Haozheng Wang, Yueheng Li Abstract Large language models struggle with representing and generating rare tokens despite their importance in specialized domains. In this study, we identify neuron structures with exceptionally strong influence on language model’s prediction of rare tokens, termed as rare token neurons , and investigate the mechanism for their emergence and behavior. These neurons exhibit a characteristic three-phase organization (plateau, power-law, and rapid decay) that emerges dynamically during training, evolving from a homogeneous initial state to a functionally differentiated architecture. In the activation space, rare token neurons form a coordinated subnetwork that selectively co-activates while avoiding co-activation with other neurons. This functional specialization potentially correlates with the development of heavy-tailed weight distributions, suggesting a statistical mechanical basis for emergent specialization. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities in learning statistical patterns of human language. However, these models consistently struggle with representing and generating rare tokens—words or phrases that appear infrequently in training data [ 17 , 40 , 21 ] . This poses significant challenges for both basic language modeling and specialized domain applications, where rare but critical information often resides in the long tail of the distribution. This challenge stems from the power-law distributions inherent in natural language [ 41 , 38 ] , with a significant portion of linguistic phenomena appearing with extremely low frequency [ 7 , 16 ] . Recent work has highlighted how this challenge can lead to collapse when models are trained on synthetic data that either truncates or narrows the tail of the distribution [ 10 , 15 , 4 ] . While several external methods have been proposed to address this limitation—such as retrieval-augmented generation [ 19 ] , in-context learning [ 11 ] , and non-parametric memory mechanisms [ 5 ] —the fundamental question remains: do LLMs develop internal mechanisms specialized for processing rare tokens during pre-training? This question draws inspirations from human language acquisition, where children demonstrate remarkable "fast mapping" abilities—learning new words after minimal exposure—from as young as 12 months of age [ 8 , 23 ] . Cognitive neuroscience explains this capability through the Complementary Learning Systems (CLS) theory [ 26 , 28 ] , which posits that the brain employs two distinct neural systems: a neocortical system for gradual learning of distributed representations, and a hippocampal system specialized for rapid encoding of specific experiences, including rare events [ 18 , 32 ] . This biological specialization enables humans to effectively learn from both statistical regularities and singular experiences. Recent advances in mechanistic interpretability have developed various techniques for analyzing individual neurons within transformer models. While studies have revealed that neurons encode interpretable features ranging from syntactic relationships [ 22 , 12 ] to semantic concepts [ 14 , 6 ] , most work has focused on common patterns rather than specialized mechanisms for rare events. Notable exceptions include work Stolfo et al. [33] discovering neurons that modulate token logits proportionally to frequency. In this study, we focus on decoder-only Transformer-based models and extend their work to focus on rare tokens and investigate how individual neurons in the final MLP layer of transformer-based language models spontaneously specialize in processing these tokens during training. Our analysis reveal three key findings: (i) LLMs develop dedicated "rare token neurons" that disproportionately impact the prediction of infrequent tokens; (ii) These specialized neurons emerge through distinct phase transitions during training, evolving from a homogeneous initial state to a functionally differentiated architecture; (iii) The emergence of specialized neuron groups correlates with the development of heavy-tailed weight distributions, suggesting a statistical mechanical basis for functional specialization. 2 Background 2.1 Transformer architecture In this study, we focus on the Multi-Layer Perceptron (MLP) sublayers. Given a normalized hidden state x\in\mathbb{R}^{d_{\text{model}}} from the residual stream, the MLP transformation is defined as: \text{MLP}(x)=W_{\text{out}}\phi(W_{\text{in}}x+b_{\text{in}})+b_{\text{out}}, (1) where W_{\text{in}}\in\mathbb{R}^{d_{\text{mlp}}\times d_{\text{model}}} and W_{\text{out}}\in\mathbb{R}^{d_{\text{model}}\times d_{\text{mlp}}} are learned weight matrices, and b_{\text{in}},b_{\text{out}} are biases. The nonlinearity \phi is typically a GeLU activation. We refer to individual entries in the hidden activation vector \phi(W_{\text{in}}x+b_{\text{in}}) as neurons , indexed by their layer and position (e.g., <layer>.<index> ). The activations n represent post-activation values of these neurons. We selected the last layer as it directly projects into the unembedding matrix that produces token probabilities, which creates a computational bottleneck where feature integration must occur [ 37 ] . 2.2 Heavy-Tailed Self-Regularization (HT-SR) Theory Heavy-Tailed Self-Regularization (hereafter HT-SR ) theory offers a spectral lens on neural network generalization [ 24 , 25 , 20 , 9 ] . Specifically, consider a neural network with L layers, let W_{i} denote a weight matrix extracted from the i -th layer, where W_{i}\in\mathbb{R}^{m\times n} and m\geq n . We define the correlation matrix associated with W_{i} as: X_{i}:=W_{i}^{\top}W_{i}\in\mathbb{R}^{n\times n}, which is a symmetric, positive semi-definite matrix. The empirical spectral distribution (ESD) of X_{i} is defined as: \mu_{X_{i}}:=\frac{1}{n}\sum_{j=1}^{n}\delta_{\lambda_{j}(X_{i})}, where \lambda_{1}(X_{i})\leq\cdots\leq\lambda_{n}(X_{i}) are the eigenvalues of X_{i} , and \delta is the Dirac delta function. The ESD \mu_{X_{i}} represents a probability distribution over the eigenvalues of the weight correlation matrix, characterizing its spectral geometry. HT-SR theory proposes that successful neural network training exhibits heavy-tailed spectral behavior in the ESDs of certain weight matrices, due to self-organization toward a critical regime between order and chaos. Such heavy-tailed behavior is captured by various estimators, and particularly informative among which is the power-law (PL) exponent \alpha_{\text{Hill}} , who estimates the tail-heaviness of the eigenvalue distribution. Low values of \alpha_{\text{Hill}} (typically \alpha<2 ) indicate heavy-tailed behavior, often interpreted as signs of functional specialization and self-organized criticality [ 39 ] . A formal definition of \alpha_{\text{Hill}} and the associated estimation procedure is provided in Section 3.4 . 3 Rare Token neuron analysis framework 3.1 Rare Token Neuron Identification Inspired by prior work on confidence-regulation neurons [ 33 ] , we hypothesize that certain neurons in language models functionally specialize for modulating token-level probabilities—particularly for rare tokens that appear infrequently in the training corpus. From a theoretical perspective, such specialization aligns with principles of sparse coding [ 27 ] and the information bottleneck framework [ 34 ] , where neural capacity is selectively allocated to maximize efficiency under uncertainty. Ablation methodology To investigate the functional specialization hypothesis, we perform targeted ablation experiments on multiple language models, such as the Pythia model families [ 3 ] , with intermediate checkpoints and training set available (The Pile [ 13 ] ). Following the intervention approach of Stolfo et al. [33] , we assess neuron-level influence by performing **mean ablation** experiments, that is, fixing a specific neuron’s activation to its mean value over a reference dataset. Formally, let i\in\{1,2,\dots,d_{\text{mlp}}\} index a neuron in the MLP layer, and let n_{i}\in\mathbb{R} denote its activation. For a given input x\in\mathcal{X} , let x represent the final hidden state (i.e., the output of the last transformer block). The mean-ablated hidden state \tilde{x}^{(i)} is then given by: \tilde{x}^{(i)}=x+(\bar{n}_{i}-n_{i})w^{(i)}_{\text{out}}, (2) where \bar{n}_{i} is the mean activation of neuron i across a reference subset of inputs, and w^{(i)}_{\text{out}} is the corresponding output weight vector. Quantifying neuron influence To quantify the influence of each neuron i , we compute the Neuron Effect metric, defined as the expected absolute change in token-level loss

Read the complete text