Quiet Feature Learning in Algorithmic Tasks

Prudhviraj Naidu, Zixian Wang, Leon Bergen, Ramamohan PaturiDepartment of Computer ScienceUC San DiegoSan Diego, CA 92093, USA {prnaidu,ziw081,lbergen,rpaturi}@ucsd.edu · 2025

Read Quiet Feature Learning in Algorithmic Tasks by Prudhviraj Naidu, Zixian Wang, Leon Bergen, Ramamohan PaturiDepartment of Computer ScienceUC San DiegoSan Diego, CA 92093, USA {prnaidu,ziw081,lbergen,rpaturi}@ucsd.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03997.txt.

Introduction

Prudhviraj Naidu, Zixian Wang, Leon Bergen, and Ramamohan Paturi reported in 2025 that language models learn hidden skills before their scores improve. The catalog creator field is garbled with affiliation and email text. Training curves for algorithmic tasks showed long flat stretches, then sudden drops in loss, against standard scaling expectations.

Probing the models' inner states during the flat stretches revealed quiet features: partial computations that alone do not help the output. Removing individual features broke task performance, proving they are needed. The team argue that real learning can hide under a flat loss curve. Watching loss alone misreads training, and richer checks on inner representations are needed to see progress.

Catalog record

CREATORPrudhviraj Naidu, Zixian Wang, Leon Bergen, Ramamohan PaturiDepartment of Computer ScienceUC San DiegoSan Diego, CA 92093, USA {prnaidu,ziw081,lbergen,rpaturi}@ucsd.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quiet Feature Learning in Algorithmic Tasks Prudhviraj Naidu, Zixian Wang, Leon Bergen, Ramamohan PaturiDepartment of Computer ScienceUC San DiegoSan Diego, CA 92093, USA {prnaidu,ziw081,lbergen,rpaturi}@ucsd.edu Abstract We train Transformer-based language models on ten foundational algorithmic tasks and observe pronounced phase transitions in their loss curves that deviate from established power-law scaling trends. Over large ranges of compute, the validation loss barely improves, then abruptly decreases. Probing the models’ internal representations reveals the learning of quiet features during the stagnant phase, followed by sudden acquisition of loud features that coincide with the sharp drop in loss. Our ablation experiments show that disrupting a single learned feature can dramatically degrade performance, providing evidence of their causal role in task performance. These findings challenge the prevailing assumption that next-token predictive loss reliably tracks incremental progress; instead, key internal features may be developing below the surface until they coalesce, triggering a rapid performance gain. 1 1 footnotetext: These authors contributed equally to this work. 1 Introduction Understanding how and when large language models acquire new capabilities has become an important question in the study of deep learning. While language models have demonstrated remarkable performance across a broad range of tasks, the precise mechanisms driving their improvements remain unknown. Recent discussions of “emergent abilities” – where larger-scale models outperform baselines abruptly, even though smaller-scale counterparts exhibit little improvement – have led to debate over whether such phenomena are genuine or artifacts of measurement ( Wei et al. 2022 ; Ganguli et al. 2022 ; Schaeffer et al. 2023 ) . Questions about emergent abilities are closely tied to the observation of scaling laws in model training ( Kaplan et al. 2020 ; Ruan et al. 2024 ; Henighan et al. 2020 ; Dubey et al. 2024 ; OpenAI 2023 ) . These scaling laws typically show a smooth, power-law relationship between compute and model performance. However, most empirical demonstrations of these laws derive from heterogeneous data and tasks, leaving open the possibility that “averaging out” many distinct learning behaviors masks more abrupt transitions that occur for individual skills or subtasks. We focus on ten foundational algorithmic problems spanning various input types. These algorithmic tasks have precisely defined solutions, making it straightforward to identify clear success criteria, isolate the specific features or subtasks the model must learn, and ensure that improvements cannot be attributed to memorization or partial heuristics. These tasks allow us to investigate fine-grained learning phenomena which might otherwise be obscured by heterogeneous data. Our key findings include: 1. Phase transitions in scaling laws : We observe two distinct phases in scaling laws across tasks and task lengths. In the “ slow phase ,” loss improves minimally or remains flat. Then, loss drops rapidly (“ fast phase ”). We refer to the change between these two phases as phase transition. We observe this phase transition when we vary model size & training dataset size, fix model size & vary training dataset size and within individual training runs. 2. Quiet features precede phase transitions : Models learn meaningful internal representations during the slow phase, but these features do not yet yield noticeable gains in the output loss (we call these “quiet features”). Ablating them severely degrades performance, demonstrating they are causally related to the eventual sharp drop in loss. 3. Lack of strict ordering by computational dependence : The emergence of “quiet” vs. “loud” features (learnt during the fast phase) does not always mirror the dependency graph of the underlying algorithm. Some pairs of features with ordering dependencies appear to emerge simultaneously. These findings challenge the assumption that improvements in loss necessarily coincide with improvements in feature representations. Instead, substantive internal reorganization may occur below the surface, revealing itself only at discrete points during training. The rest of this paper is organized as follows: Section 2 reviews related work in scaling laws, emergent abilities, and algorithmic learning. Section 3 describes our experimental methodology and presents our observations of phase changes across tasks and task lengths. Section 4 introduces our feature analysis framework and demonstrates how quiet and loud features evolve during training. Finally, Section 5 discusses the broader implications of our findings and suggests directions for future research. 2 Related Work 2.1 Scaling Laws Hestness et al. 2017 observed scaling dataset size and deep neural network model size led to a predictable decrease in generalization error for neural machine translation, language modeling, image classification and speech recognition. Kaplan et al. 2020 , Hoffmann et al. 2022 observed predictable relationships between training compute and language modeling loss. Henighan et al. 2020 extended this work for generative models across modalities: image, video, multimedia image-text and math. They demonstrated classification loss and error rates predictably decreased on downstream image classification tasks. Chen et al. 2021 studied language model performance on coding. They observed a predictable relationship between language modeling loss on a held out code corpus and model size. 2.2 Predicting LLM abilities Ganguli et al. 2022 , Wei et al. 2022 demonstrated one could not predict large language models capabilities from capabilities of small language models. However, Schaeffer et al. 2023 , OpenAI 2023 , Ruan et al. 2024 , Dubey et al. 2024 showed this was due to choice of metrics and large language models capabilities could be predicted from capabilities of small language models. 2.3 Proposed explanations for scaling laws Michaud et al. 2023 proposed neural networks learn discrete skills called ”quanta.” They argue that there is a strict ordering, which they called Q sequence, in which quanta must be learned. They proposed the frequencies of these quanta follow a power law, leading to the power law relationship observed by Kaplan et al. 2020 and others. Hutter 2021 propose that the relationship between the error rate and dataset size is guided by the distribution of features in the data. They show a Zipfian distribution results in power law scaling. 2.4 Grokking In grokking ( Power et al. 2022 , Nanda et al. 2023 , Varma et al. 2023 ), a model trained for many epochs quickly memorizes the training set (thus achieving high training accuracy early) but only later learns a more generalizing representation, causing a sudden jump in test accuracy. In contrast, our phase transitions occur during single epoch training, with both training and test performance starting off poor. The model gradually learns “quiet” internal features before suddenly mastering the entire task. Hence, our model does not show a transition from memorization to generalization. 2.5 Progress Measures Several previous works have identified measures which track progress toward the final, fully-generalizing solution, even when the test loss shows no improvement. Barak et al. 2022 propose a metric for measuring similarity of network weights in the context of sparse parity, and demonstrate that this metric continuously improves throughout training, including prior to measurable improvement in generalization performance. Nanda et al. 2023 propose a different metric on network weights in the context of modular arithmetic grokking, and demonstrate that this metric improves before the phase transition. Mallinar et al. 2024 propose tracking features using Average Gradient Outer Product (AGOP) for Recursive Feature Machines. While these prior works demonstrate increasing similarity between network weights and a generalizing solution prior to the emergence of improved generalization behavior, they do not

Read the complete text