Hidden Breakthroughs in Language Model Training

Sara Kangaslahti, Elan Rosenfeld, Naomi Saphra · 2025

Read Hidden Breakthroughs in Language Model Training by Sara Kangaslahti, Elan Rosenfeld, Naomi Saphra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15872.txt.

Introduction

Sara Kangaslahti and co-authors hunt for hidden jumps in skill as language models train. The work dates to 2025. Loss curves run smooth for most of training, so sharp drops stand out as skill jumps. The team argues like jumps occur oft but hide in the lumped loss score.

Their method splits loss shifts along set paths in the low-rank train space. Groups of samples with like shifts then stand out as clusters. Tests on toy math and on word tasks showed the clusters map to clear skill jumps. The finds hint such jumps can serve as a no-label lens on how models learn.

Catalog record

CREATORSara Kangaslahti, Elan Rosenfeld, Naomi Saphra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15872

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hidden Breakthroughs in Language Model Training Sara Kangaslahti, Elan Rosenfeld, Naomi Saphra Abstract Loss curves are smooth during most of model training, so visible discontinuities stand out as possible conceptual breakthroughs. Studying these breakthroughs enables a deeper understanding of learning dynamics, but only when they are properly identified. This paper argues that similar breakthroughs occur frequently throughout training but they are obscured by a loss metric that collapses all variation into a single scalar. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along arbitrary bases of the low-rank training subspace. We use our method to identify clusters of samples that share similar changes in loss during training, disaggregating the overall loss into that of smaller groups of conceptually similar data. We validate our method on synthetic arithmetic and natural language tasks, showing that POLCA recovers clusters that represent interpretable breakthroughs in the model’s capabilities. We demonstrate the promise of these hidden phase transitions as a tool for unsupervised interpretability. 1 Introduction As large language models train, various internal structures develop during abrupt phase transitions. These sudden drops in loss reveal the formation of mechanisms for in-context learning [ 25 ] , natural language grammar [ 6 ] , hierarchical generalization [ 21 ] , and many other concepts [ 19 , 16 , 27 , 1 ] . However, the loss curve as a whole remains stubbornly smooth. As a result, these momentary conceptual breakthroughs are treated as isolated curiosities, while the majority of training behavior is considered predictable. This work shows that in fact, a model undergoes many breakthroughs during training, most which are concealed when averaging all data into a single loss curve. Instead, we divide up the loss curve in two different ways to find hidden breakthroughs. First, we disaggregate the aggregate loss into losses on individual examples. By clustering the individual loss curves, we identify subsets of data that experience synchronized changes in loss, implying that they rely on the same conceptual breakthrough. However, any individual example might benefit from multiple breakthroughs; such an example may undergo changes synchronized with different data subsets at different times. Furthermore, distinct concepts might appear simultaneously, erroneously merging their data clusters. This means we may have to identify multiple separate breakthroughs which affect the loss curve of a single example. To disentangle these effects for a single sample, we separate the optimization space into specific gradient directions. When the loss changes during training, it is the result of movement across all parameters in a high-dimensional space. We decompose this loss change from an exact trajectory in the full-rank parameter space into a collection of movements along each dimension. By analyzing these loss curves along specific basis vectors, we identify conceptual breakthroughs that rely on particular directions of movement. The latter analysis permits further granularity in clustering data, as final performance on an individual example may rely on multiple conceptual breakthroughs, each corresponding to a particular linear direction in training. In summary: • We introduce a modified form of Loss Change Allocation [ 15 ] called Projection Oriented Loss Change Allocation (POLCA) to measure changes in loss due to parameter adjustments in arbitrary directions during training (Section 3.2 ). • We show that some learned concepts can be identified by clustering exact loss, while others cannot (Section 4.2 ). • Using POLCA, we extend our cluster analysis to identify conceptual breakthroughs in a restricted gradient subspace that are obscured in the exact loss curves. We automatically identify specific concepts learned during breakthroughs in both synthetic (Section 4 ) and natural language settings (Section 5 ). 2 Background: How much can we learn from learning dynamics? Figure 1 : A smooth loss function may change sharply for a particular direction or data subset. POLCA works by decomposing and disaggregating the loss to discover these sharp changes. Left: Loss L(x;\theta) changes as the parameter setting \theta moves in a low-rank training subspace. The loss is sigmoidal on each axis, with differently timed inflections along basis vectors {\color[rgb]{1,0.52,1}b_{1}} and {\color[rgb]{1,0.58,0}b_{2}} . These breakthroughs disappear in the smooth sum of the sigmoids which represents the exact loss. Right: The average of sigmoidal functions—including loss along basis vectors {\color[rgb]{1,0.52,1}b_{1}} and {\color[rgb]{1,0.58,0}b_{2}} —elides individual breakthroughs. The more differently-timed breakthroughs underlie the loss, the more hidden each breakthrough is. Various loss phase transitions have been interpreted as conceptual breakthroughs. But why expect additional interpretable breakthroughs to underlie periods of undifferentiated, gradual model improvement? Our approach is justified by the nature of the loss surface’s complexity, illlustrated by Figure 1 , in which a smooth curve emerges by eliding phase transitions from each dimension. Why expect multiple phase transitions? A very early phase transition is to be expected early in training after a brief memorization stage [ 30 ] . In this sense, the most celebrated breakthroughs are not the first phase transitions in their training curves. Some breakthroughs even depend on earlier breakthroughs, as observed in synthetic tasks [ 1 ] and in grammar acquisition [ 6 ] . If one concept depends on another, each must appear at a different timestep, requiring multiple phase transitions. Multiple phase transitions can also come from differences in gradient scale along different directions. Ma et al. 2022 even attributed the early edge-of-stability phase transition [ 13 , 8 ] to multiscale structure of the loss surface and, furthermore, noted that this multiscale structure emerges at the range where models become singular : their loss lacks a quadratic approximation in terms of model parameters, creating conditions for phase transitions under Singular Learning Theory [ 33 , 34 , 32 ] . They argued that this structure is the product of both nonuniform data and nonconvex objectives , respectively justifying the disaggregation and decomposition which we apply to interpret training dynamics. Why disaggregate the aggregate loss? We track learning on training datapoints and subpopulations, rather than the whole training set, because relevant skills can be acquired at different rates [ 3 , 7 ] . Individual samples thus exhibit changes in loss out of line with the monotonic average trend [ 36 , 29 ] . In full-batch gradient descent, Cohen et al. 2022 identified non-monotonicity arising from oscillation about the maximum Hessian eigenvector. Rosenfeld and Risteski 2024 demonstrated that these oscillations occur across different axes for different samples and identified the primary cause: surprisingly human-interpretable semantic features. Even when the loss seems stable, performance can oscillate on edge cases until the model develops relevant capabilities [ 28 , 4 ] . We hypothesize that oscillation represents competing skills that are relevant to different subsets of data. To test this hypothesis, and to interpret the meaning of these directions, we disaggregate the loss into clusters with similar dynamics. Why decompose the exact loss? Our POLCA decomposition addresses what Michaud et al. 2024 called polygenic scaling effects—samples which combine multiple skills and therefore exhibit breakthroughs at multiple scales. If we assume that a specific skill is enabled by movement along a particular skill’s basis vector, then the loss change attributed to movement along that vector will accelerate at the moment the skill is acquired, for every sample that requires that skill. In this manner, the sample transitions

Read the complete text