Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

Juliana Li, Diya Sreedhar · 2026

Read Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining by Juliana Li, Diya Sreedhar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26050.txt.

Introduction

Juliana Li and Diya Sreedhar report a strange training phenomenon they call natural ungrokking. A small language model learns a pronoun-gender rule early in pretraining. Cued with a girl's name, it picks the matching pronoun on held-out probes. A few thousand steps later, the skill vanishes. The rule's evidence never left the training data. Loss curves show no sign of the reversal.

One corpus statistic predicts which rules survive. Rules live or die by how often the training stream shows them winning. Data volume only deepens the fall of doomed rules. A rival surface pattern out-competes the rule inside the model. Their internal margin crosses zero within a hundred steps of the visible collapse. Public Pythia checkpoints show the same dynamics at several scales. Control turns out lopsided. Flooding the data with counter-evidence kills a rule on demand. Flooding it with support cannot revive one.

Catalog record

CREATORJuliana Li, Diya Sreedhar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26050

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining Juliana Li, Diya Sreedhar Abstract Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl’s name (“ Sue cried because ”), it resolves the next pronoun to she , generalizing to held-out probes ( 0.94 0.94 by step 925). By step 3,500 the same model scores near zero on the same probes, although the rule’s evidence is still in the training data. We call this within-run reversal natural ungrokking : the corpus decides, with no trace in the loss curve, which learned rules a model keeps. Which rules survive is predictable from one corpus statistic: how often the training stream shows the rule winning. Across un-intervened runs (two corpora, three data budgets, three seeds), support frequency decides a rule’s fate; the data-to-parameter ratio only modulates how deeply a doomed rule falls. The same emerge-then-collapse training dynamics appear in public Pythia checkpoints, collapse depth ordered by model scale as predicted. The forgetting is a displacement: a competing surface pattern out-competes the rule, and the log-probability margin between them crosses zero within 100 training steps of the behavioral collapse. Control over this fate is asymmetric: the same edit that destroys a rule on demand cannot restore it. Flipping support to counter-evidence in place kills the rule with monotone dose-response in two unrelated rules; but injecting support back—even to 450 × 450\times the level that naturally sustains it—buys no recovery. Every confirmatory threshold and prediction was pre-registered before the data it governed was read. 1 Introduction Figure 1 : The focal pronoun-gender rule emerges and then collapses under web pretraining, and an internal margin crosses zero as it does. (a) Held-out conflict accuracy. On TinyStories (green: seed mean, min–max band) the rule survives at ceiling; on web the bold vermillion trace is the seed the abstract quotes ( 0.94 by step 925, gone by the end), with the two faint traces rising and collapsing the same way. Dotted lines are each corpus’s agree-condition control, which keeps climbing through the collapse: the construction stays solved and only the rule is lost. Grey dotted marks chance. (b) Contrast margin \mathrm{CM} ( Section 4.2 ), the model’s preference for the rule over the surface default, one strip per instrument-valid web seed; the vertical line marks behavioral collapse onset, the dot the \mathrm{CM} zero crossing. The two coincide in both seeds (steps 2{,}800/2{,}800 and 2{,}900/3{,}000 ). The third web seed fails the instrument check ( Section 3 ) and is omitted; its crossing is directionally consistent. TinyStories margins stay off scale at +2 to +12 nats. Within a single pretraining run, a language model can lose a capability it had already acquired. The case this paper dissects is the pronoun-gender rule in an 11.5M-parameter model training on web text: cued with a girl’s name, the following pronoun should resolve to she . A conflict probe item is the prefix “ Sue cried because ”, scored on the continuation she against he ; the corpus-wide prior favors he , the name cue demands she . Its agree-condition counterpart, “ Tom cried because ”, aligns cue and prior and acts as the control. The model scores 0.94 on held-out conflict items by step 925 and near 0.00 by steps 3,500–4,400 of the same run, while still scoring 1.00 when rule and prior agree and the rule’s evidence stays in the stationary stream. No training data was removed, and no distribution shifted. The model acquired the rule and then stopped applying it ( Figure 1 a). We call this within-run reversal natural ungrokking . A capability sighted at a mid-training checkpoint is no guarantee it will exist in the final model, but its support frequency in the corpus is predictive of whether it survives. Data filtering that thins a rare rule’s support can doom that rule without deleting a single example of it. Continual pretraining on a shifted mix can silently ungrok capabilities the base model had. In every case the failure is invisible to loss curves, because the model keeps the construction and abandons only the rule. This paper makes three contributions. Capabilities are cheap to destroy and hard to restore. Editing a rule’s evidence in place, flipping it to counter-evidence at a chosen rate with token counts and every other corpus statistic held fixed, destroys the rule with strictly monotone dose-response. A second, unrelated rule ( a / an allomorphy) replicates the kill on a five-dose ladder built inside one corpus, dialing final accuracy monotonically from 0.96 to 0.00 while unrelated capabilities hold at baseline; in both rules, the scored blind predictions about kill direction all came out correct. The same knob fails in reverse—we injected support back into the collapsed corpus at up to 3\times the TinyStories support density, reaching post-injection rule:prior ratios far above the surviving TinyStories cell; the mechanistic margin moved, but no run produced a behavioral recovery that passed its controls ( Section 4.3 ). A rule’s fate follows a support-frequency law. Whether a rule survives to the end of training is decided by how often its supporting evidence appears in the corpus; the unique-data-to-parameter ratio D/N modulates how deeply a doomed rule falls but never flips its fate ( Section 4.1 ). We establish this across a grid of corpora, data budgets, and seeds. The same rise-and-fall reproduces in the smaller public Pythia checkpoints, with collapse depth following the predicted scale order, and the verdicts transfer to an out-of-distribution probe set. The loss is displacement. The collapsing rule loses a competition with a strengthening surface pattern, here the corpus-wide preference for he , while the construction it lives in stays solved. A contrast margin between rule and prior crosses zero at the behavioral collapse step, and its final sign largely separates recovered from displaced cells across the grid, with a boundary case discussed below ( Section 4.2 ). Every threshold, metric, and directional prediction here was registered in advance ( Section 3 ), with the full scoreboard in App. A . Predicting a rule’s fate works; predicting where on the frequency axis the survival boundary falls is the open problem this paper hands to future work (App. B ). Code, configs, probe batteries, and the frozen registration document are released. 1 1 1 https://github.com/lijuliana/Natural-Ungrokking 2 Related Work Grokking is delayed generalization ( Power et al. 2022 ; Nanda et al. 2023 ) , with phase diagrams drawn over data size and hyperparameters ( Liu et al. 2023 ; Liu et al. 2022 ) . The nearest dynamical relative of our subject is ungrokking ( Varma et al. 2023 ) , where generalization recedes when the dataset shrinks below a critical size between runs; our capabilities rise and fall within a single run on stationary text, which is the kinship and the difference that natural ungrokking names. Transience itself has precedent: Singh et al. 2023 show emergent in-context learning can fade under continued training; we observe the analogous fate for in-weights linguistic rules, give it a frequency law, and steer it with corpus edits. Chen et al. 2024 document sudden transitions in syntax acquisition and control them through the training data; we follow the same observational-to-causal arc

Read the complete text