Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention

Jing Huang , a {}^{\raisebox{0.5pt}{\hskip 1.42262pt\includegraphics[height]{style/goodfire_logo_small.png}},a} Daniel Wurgaft , a {}^{\raisebox{0.5pt}{\hskip 1.42262pt\includegraphics[height]{style/goodfire_logo_small.png}},a} Rachit Bansal b Laura Ruis c Naomi Saphra b David Alvarez-Melis b Andrew Lampinen d Christopher Potts a Ekdeep Singh Lubana a Stanford University b Kempner Institute at Harvard University c MIT d Anthropic · 2026

Read Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention by Jing Huang , a {}^{\raisebox{0.5pt}{\hskip 1.42262pt\includegraphics[height]{style/goodfire_logo_small.png}},a} Daniel Wurgaft , a {}^{\raisebox{0.5pt}{\hskip 1.42262pt\includegraphics[height]{style/goodfire_logo_small.png}},a} Rachit Bansal b Laura Ruis c Naomi Saphra b David Alvarez-Melis b Andrew Lampinen d Christopher Potts a Ekdeep Singh Lubana a Stanford University b Kempner Institute at Harvard University c MIT d Anthropic (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29548.txt.

Introduction

Jing Huang and eight colleagues explain why larger models learn tasks smaller models miss, from data. An argument suggests bigger models capture data parts smaller ones fail on, even with unlimited training. In mixtures, smaller models spend neurons on frequent or simple tasks and fail rare, complex ones, even when a solution exists. Larger models ease this bottleneck because common gradients weaken once covered, leaving rare-task features intact. OLMo pretraining from four million to four billion parameters mirrors this, with only larger models learning infrequent, complex tasks.

Catalog record

CREATORJing Huang , a {}^{\raisebox{0.5pt}{\hskip 1.42262pt\includegraphics[height]{style/goodfire_logo_small.png}},a} Daniel Wurgaft , a {}^{\raisebox{0.5pt}{\hskip 1.42262pt\includegraphics[height]{style/goodfire_logo_small.png}},a} Rachit Bansal b Laura Ruis c Naomi Saphra b David Alvarez-Melis b Andrew Lampinen d Christopher Potts a Ekdeep Singh Lubana a Stanford University b Kempner Institute at Harvard University c MIT d Anthropic
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85851
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29548

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention Jing Huang , a {}^{\raisebox{0.5pt}{\hskip 1.42262pt\includegraphics[height]{style/goodfire_logo_small.png}},a} Daniel Wurgaft , a {}^{\raisebox{0.5pt}{\hskip 1.42262pt\includegraphics[height]{style/goodfire_logo_small.png}},a} Rachit Bansal b Laura Ruis c Naomi Saphra b David Alvarez-Melis b Andrew Lampinen d Christopher Potts a Ekdeep Singh Lubana a Stanford University b Kempner Institute at Harvard University c MIT d Anthropic Abstract Larger models learn tasks smaller models do not. What drives this phenomenon? We develop a simple phenomenological argument that power-law scaling already suggests that a larger model will be able to learn a part of the data distribution that a smaller model fails to learn, even with infinite training data. To validate this claim and identify its causes, we study the effects of model scaling on a synthetic setup consisting of a mixture of tasks that show monotonic scaling curves. The results point to a data-induced competition over resources (neurons). Specifically, smaller models allocate their neurons to high frequency or low complexity tasks, and so they learn solutions that perform poorly on rare and complex tasks. Moreover, this happens even when solutions capable of expressing the desired task exist. We then assess how a larger model circumvents this data-centric bottleneck, finding that it traces to a reduced interference mechanism: larger models can allocate enough resources to common tasks that the gradient updates for those tasks become weak, which means that they do not overwrite rare-task features as they slowly accumulate. Finally, to further validate these claims, we pretrain OLMo models (4M to 4B parameters) on novel tasks of varying frequency and complexity. The results mirror those from our synthetic data experiments: only the larger OLMo models learn the infrequent and complex tasks, and these larger models embed more task features in their representations and show less gradient interference between tasks. Overall, we offer a data-centric account of why larger models learn tasks that smaller models fail to. This helps explain why larger models are better in practice, and it can inform practical questions concerning model sizing and training data mixtures. 1 Introduction Modern machine learning is celebrated for its massive generalist models, which are capable of handling arbitrary inputs in diverse and complex environments [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] . Based on the empirical finding that larger models often excel where smaller 1 1 1 We use the terms “larger” and “smaller” informally here but develop a precise relational definition of these terms in Sec. 2 . models show random-chance performance, prior work has claimed that the ability to solve certain critical tasks only emerges in larger models [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] . Such arguments have fueled the drive towards increased scaling. However, given the large training and inference costs that large models impose, it is worth identifying precisely what marginal benefits are unlocked by larger models and whether scaling parameters is the sole way of realizing those benefits. Our argument begins from the observation that power-law scaling [ 20 , 21 , 22 ] already suggests that there is a regime in which a smaller model fails to learn parts of a data mixture that a larger model succeeds on, even under asymptotic training (Fig. 1 , Sec. 2 ). This suggests that larger models enjoy a genuine advantage that may allow them to learn task distributions that smaller models will inevitably fail to learn within the same training setup. Importantly, this is not an argument that larger models are simply more sample efficient [ 23 , 24 , 25 , 26 , 18 , 27 , 28 , 29 ] , but rather that smaller models suffer from a more fundamental limitation even under infinite training regimes. Figure 1 : Learning a part of the distribution requires model scaling. Compare the loss curves for compute-optimal scaling with the one following an infinite resource regime (labeled asymptotic). The region labeled purple denotes the amount of loss both a smaller model with N_{s} parameters and a larger model with N_{l} parameters are able to achieve with respect to a random baseline under finite resources. We call loss reduction accessible to the smaller model under infinite compute, but that a larger model would get at in a more resource efficient manner (i.e., under finite compute), learnable via data scaling . If there remains a part of the loss that is achieved by the larger model under finite resources, but that a smaller model even under asymptotic data scaling is unable to reach, then we call this part learned via model scaling . This part of the distribution is explained by a larger model by virtue of its larger size. To validate this prediction and identify its causes, we analyze a setting involving a mixture of regression tasks. In this, we are inspired by much recent work using toy tasks to pinpoint the effects of scaling [ 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 ] . Furthermore, all of the individual tasks in our setting are learnable by the models under consideration, capturing the idea that tasks smaller models fail to learn can still be instilled into them via post-training [ 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 ] . Correspondingly, mere expressivity notions are not the issue; instead, the question concerns the ability of these models to learn complex task distributions from data. These experiments lead to two key findings, as described below. First, scaling enables learning rare and complex tasks (Sec. 3.1 ). Our experimental setting defines controlled manipulations of task frequency and complexity. We present an analytic argument that only larger models will (on average) learn the rare and complex tasks present in this setting, and we verify this analysis experimentally (Fig. 2 ). Second, reduced competition for resources enables learning rare and complex tasks (Sec. 3.2 ) . Here, we extend our formal analysis to show that, upon observation of samples from a rare task, model parameters update, but only larger models, by virtue of having more parameters and hence less gradient interference, are able to retain memory of a previously observed batch of data from a rare task. Thus, when the next batch of rare-task data comes in, the larger model builds on its prior knowledge, which ultimately leads to success despite the impoverished learning signal. In contrast, the smaller model is forced to start from scratch and consequently fails. We again verify these findings experimentally in our regression setting (Figs. 3 and 4 ). Finally, we validate the above theoretical arguments in real LLMs (Sec. 4 ). Specifically, we pretrain OLMo models (4M to 4B parameters) on the Dolma v1.7 corpus with completely novel tasks injected at controlled frequency. We find that only the larger OLMo models are able to learn the infrequent and complex tasks (Sec. 4.2 ). Furthermore, these OLMo models mirror our toy-task models in deeper ways: larger OLMo models have more task features embedded in their

Read the complete text