Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
Mark Rofin, Jalal Naghiyev, Michael Hahn · 2026
Read Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors by Mark Rofin, Jalal Naghiyev, Michael Hahn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14087.txt.
Introduction
Rofin, Naghiyev, and Hahn ask why transformers learn features that look useless for the next token. Some computed features seem spare for the direct task, yet they arise in training again and again. The authors trace which parts of the gradient signal give rise to such features. A new method estimates how much each part shapes a given feature. Checks on toy tasks, OthelloGPT, and a small language model support the account, and code-like domains stand out in a large pretrained model.
Catalog record
| CREATOR | Mark Rofin, Jalal Naghiyev, Michael Hahn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70107 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14087
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors Mark Rofin, Jalal Naghiyev, Michael Hahn Abstract Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal from the next-token prediction objective give rise to this phenomenon, and we propose a method to estimate the influence of those components on the emergence of specific features. After validating our approach on toy tasks, we use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small language model. Finally, we apply our framework to a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes a step toward understanding hidden features of Transformers through the lens of their development during training. Website: https://markfryazino.github.io/useless-features-iclr Code: https://github.com/Markfryazino/useless-features-iclr-code 1 Introduction Large Language Models (LLMs) are usually pretrained with the objective of next-token prediction (NTP). In this paradigm, a model learns to predict each token in a sequence given all previous tokens: in other words, it learns the distribution p(x_{t+1}\mid x_{1}\cdots x_{t}) . Thus, the model is incentivized to compute features that help predict the immediate next token. Hence, one could reasonably expect that the hidden representations at position t , computed by a model trained in this way, would contain only the information relevant for predicting x_{t+1} . On certain synthetic tasks, this was found to be true, highlighting a downside of NTP as a training objective ( Bachmann & Nagarajan 2024 ; Thankaraj et al. 2025 ) . However, a growing body of work on LLMs (and NTP-trained Transformers in general) shows that sometimes they learn much more than that. For example, Transformers reconstruct abstract features of the input text ( Templeton et al. 2024 ; Park et al. 2024 ) , infer the high-level structure of the processes generating their training data, forming ‘world models’ ( Li et al. 2023 ; Karvonen 2024 ; Shai et al. 2024 ; Jin & Rinard 2024 ; Gurnee & Tegmark 2023 ) , or implicitly predict the sequence multiple tokens ahead ( Pal et al. 2023 ; Jenner et al. 2024 ) . Motivated by these intriguing findings, we ask: How do Transformers trained for NTP learn features that don’t help in the prediction of the immediate next token? The prior work investigating learned features in Transformers mostly employed a teleological perspective: that is, features are viewed in the context of their role in the algorithms implemented by a trained model (e.g., Ameisen et al. 2025 ; Arditi et al. 2024 ). This approach is useful to find the circuits encoded in LLMs, but it doesn’t tell us much about the gradient signal that causes those circuits to develop during training. Thus, the ways of how training for NTP drives the emergence of features has been largely underexplored so far. Towards closing this gap, we develop a novel view on features learned by Transformers. Based on the structure of information flow in causally masked Transformers, we show that features can in principle be learned by three distinctive mechanisms, which we refer to as direct learning , pre-caching , and circuit sharing . The two latter ones allow the token distribution at positions >i+1 to influence the model’s representations at position i , unlocking the learning of “useless” features. Next, for a given feature, we propose an experimental method to classify it depending on which mechanism contributed the most to its development. We then use our framework to understand the learned features in Transformers trained on different data domains, including toy functions, the board game of Othello, and language. Our key contributions include: (i) A theoretically grounded explanation for why Transformers trained for NTP learn complex features that are not immediately helpful; (ii) An approach for tracing the gradient components of the NTP objective that led to the development of a given feature in a model; (iii) Novel findings obtained using the proposed framework, including an explanation of the OthelloGPT world model fragility, inspection of the role of pre-caching in text generation, and interpretation of the pre-cached features in an LLM. 2 Setup We use \mathcal{X} to denote a variable-length input space of discrete token sequences x_{1}\dots x_{n} . We view a model T_{\theta} as representing a function x_{1}\dots x_{n}\to\hat{x}_{2}\dots\hat{x}_{n+1} such that \displaystyle\hat{x}_{i+1}(x)=h_{\theta}^{L+1}(r^{L}_{i})\ \ \ \ \ \ \ \ \ r^{0}_{\theta,i}(x)=h_{\theta}^{0}(x_{i}) \displaystyle r^{k}_{\theta,i}(x)=h_{\theta}^{k}(r^{k-1}_{1}\dots r^{k-1}_{i}),\;k>1\ \ \ \ \ \ \ \ \ \ \ and r_{\theta,i}^{k}(x)\in\mathbb{R}^{d} . Here h_{\theta}^{0} and h_{\theta}^{L+1} are embedding and unembedding layers, respectively, r_{\theta,i}^{k}(x) are the values of the residual stream, and h_{\theta}^{k} are Transformer blocks. We call a learned feature any linear component of the residual stream at a specific layer and position \langle w_{i}^{k},r_{\theta,i}^{k}(x)\rangle , where a vector w_{i}^{k}\in\mathbb{R}^{d} defines the feature direction . We informally call a feature of a sequence x_{<t} NTP-useless if there exists an optimal next-token predictor for x_{t} that doesn’t compute it. Otherwise, we call that feature NTP-useful . For example, NTP-useless features can include the positioning of the board game pieces that don’t affect the set of possible next moves, or the surface properties of the sequence that are irrelevant to its continuation. The central question of our work can then be formulated as understanding how NTP-useless features emerge in Transformers. 3 Information Flow Decomposition 3.1 Gradient Decomposition We fix position i and layer k and study all information paths in the computational graph of the model, classifying them by how they relate to r_{\theta,i}^{k}(x) . We argue that the gradient training signal can flow to \theta through three types of paths, illustrated in Figure 1 . Firstly, a gradient signal can come from the immediate next-token prediction ( direct learning ). This includes all paths passing through r_{\theta,i}^{k}(x) and \hat{x}_{i+1} and represents the effect of the information encoded in r_{\theta,i}^{k}(x) on the prediction of the immediate next token. These paths are colored green in Figure 1 . We formalize this by comparing the overall gradient with a gradient after a stop-gradient operator is applied: \nabla_{\theta}L^{k}_{i\;\mathrm{(direct)}}=\nabla_{\theta}L_{i}-\nabla_{\theta}L_{i}^{\mathrm{sg}(k,i)}. (1) Secondly, the information encoded in r_{\theta,i}^{k}(x) affects the loss at positions j>i because attention heads at those positions can attend to position i . Thus, a gradient signal can come from prediction loss at future positions, after passing through one or more attention operations ( pre-caching ). This component includes the paths passing through r_{\theta,i}^{k} and \hat{x}_{j} for j>i+1 ( blue in Figure 1 ): \nabla_{\theta}L^{k}_{i\;\mathrm{(pre\text{-}cached)}}=\nabla_{\theta}\sum_{j\neq i}\left[L_{j}-L_{j}^{\mathrm{sg}(k,i)}\right]. (2) Third, there are paths that do not pass through r_{\theta,i}^{k}(x) at all. Hence, some gradient signal arises independently of r_{\theta,i}^{k}(x) . Since Transformer blocks use the same parameters to perform the computation at every position, this signal may influence the parameters computing it. We call this phenomenon circuit sharing and visualize the related paths in orange in Figure 1 : \nabla_{\theta}L^{k}_{i\;\mathrm{(shared)}}=\sum_{j}\nabla_{\theta}L_{j}^{\mathrm{sg}(k,i)}. (3) The term pre-caching is borrowed from Wu et al. 2024 , who studied it as a possible explanation for look-ahead in LLMs. We discuss the relation of our work to Wu et al. 2024 in Section 6 . Figure 1: Left: An illustration of the