Curriculum Learning for LLM Pretraining An Analysis of Learning Dynamics

Mohamed Elgaar, Hadi Amiri · 2026

Read Curriculum Learning for LLM Pretraining An Analysis of Learning Dynamics by Mohamed Elgaar, Hadi Amiri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21698.txt.

Introduction

Mohamed Elgaar and Hadi Amiri ask whether the order of training data changes how a language model learns, or only what it sees when. Curriculum learning borrows from schooling: easy first, hard later. Whether that reorders real learning or just exposure was unclear.

The team trains models of several sizes on a huge token budget under three curricula based on language development. Each is compared against random ordering, with a larger check at the top size. The answer guides data teams: if order only shuffles exposure, effort is better spent on data quality than on lesson plans.

Catalog record

CREATORMohamed Elgaar, Hadi Amiri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21698

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Curriculum Learning for LLM Pretraining An Analysis of Learning Dynamics Mohamed Elgaar, Hadi Amiri Abstract Curriculum learning changes the order of pretraining data, but it remains unclear whether it changes the learning trajectory or mainly reorders exposure over a fixed trajectory. We train Pythia models (14M--410M parameters) for 300B tokens under three linguistically motivated curricula---Age-of-Acquisition, word frequency, and Verb Variation (VV)---and compare each against Random ordering; at 1B parameters we compare Random and VV. Across orderings, training follows a shared sequence of latent phases, while curricula mainly change within-phase data exposure. In smaller models (up to 160M parameters), Random ordering exhibits higher gradient noise and stronger late-training output-head spectral saturation, alongside lower final accuracy; curricula reduce both effects at matched compute. At larger scales, saturation differences are smaller and curriculum gains shrink. We formalize the link between difficulty pacing and optimization stability in an idealized analysis based on gradient-variance control, and our results point to a practical takeaway: curricula help by stabilizing within-phase optimization rather than by creating new phases. 1 1 1 Code, data, and checkpoints will be made available. Figure 1 : Overview of the mechanism studied in this work. In smaller, capacity-constrained models, random data ordering can produce higher gradient noise scale, coinciding with late-stage spectral saturation of the output head. Linguistically motivated curricula (Age-of-Acquisition, Frequency, and Verb Variation) reduce gradient noise and are associated with improved spectral stability. 1 Introduction Pretraining large language models requires substantial compute, often on the order of hundreds of petaflop-days per run ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) . Accordingly, modern practice typically trains for a single epoch over massive corpora: revisiting data is inefficient when new data are abundant ( Komatsuzaki 2019 ) . Compute-optimal scaling laws similarly treat the budget as the total number of tokens consumed from a unique corpus ( Hoffmann et al. 2022 ) . In this single-pass setting, data ordering is consequential: each sample is encountered exactly once. Curriculum learning—presenting training examples in a structured order, often easy-to-hard—offers a way to shape this process ( Bengio et al. 2009 ; Elman 1993 ) . One motivation is that learning may be easier when foundational patterns come before rarer or more complex ones. Despite this intuition, results on curriculum learning for LLM pretraining are mixed. Some studies report gains at fixed compute budgets ( Fan & Jaggi 2023 ) , whereas others find diminishing or negligible effects at scale ( Campos 2021 ) . One reason is that it remains unclear which aspects of training dynamics curricula affect. Curricula could either change the learning trajectory—the sequence of internal states a model traverses during training—or primarily change which data the model sees at each stage without altering the underlying dynamics. To design curricula that improve training consistently, we need to separate these possibilities. This distinction matters most for smaller models, where capacity constraints make the interaction between data ordering and learning dynamics particularly consequential. These models face structural limits such as the softmax bottleneck ( Godey et al. 2024 ; Michaelov & Bergen 2023 ) —where a low-rank output head cannot represent the high-rank distributions required for language modeling. As training progresses, this mismatch can cause late-stage degradation, and its interaction with data ordering may not be apparent from end-point aggregates such as final loss or benchmark accuracy. We address these questions with a systematic study of data ordering in LLM pretraining. We train Pythia models (14M–410M parameters) for 300B tokens under three linguistically motivated curricula—Age-of-Acquisition, word frequency, and Verb Variation (VV)—and compare them against Random ordering; for 1B parameters, we compare Random and VV due to compute constraints. Beyond aggregate metrics, we analyze learning dynamics in two ways. First, we perform latent phase analysis by fitting Hidden Markov Models to training trajectories. Second, we track optimization diagnostics: the gradient noise scale ( McCandlish et al. 2018 ) , a measure of gradient signal-to-noise ratio, and singular entropy ( Godey et al. 2024 ) , which quantifies spectral saturation of the output head. We summarize the mechanism in Figure 1 . Across all orderings, training follows a shared sequence of latent phases. Curricula mainly change which data appears within each phase. The effect is strongest at small scales: for smaller models (14M–70M), Random ordering exhibits higher gradient noise scale; for models up to 160M, it also shows stronger late-training spectral saturation. Curricula reduce both effects and can improve accuracy at matched compute. At larger scales (410M+), these differences diminish and curriculum gains become marginal. This paper makes the following contributions: • We show that pretraining trajectories under different data orderings share a consistent latent phase structure. Using a joint HMM analysis across orderings (illustrated for the 14M model, and with multiple random seeds for small models), we find that curricula mainly reshape within-phase data exposure rather than changing the phase sequence. • We provide a theoretical framework formalizing curriculum learning as a stability mechanism ( Theorem 3.2 ): by controlling the difficulty of data encountered over training, curricula can bound gradient variance and reduce late-stage optimization instability. • We demonstrate empirically that curricula reduce gradient noise scale and output-head spectral saturation in smaller models. These reductions align with weaker late-stage effects that we associate with the softmax bottleneck, and they diminish at larger scales where capacity is sufficient to avoid saturation. • We report experiments spanning five model sizes (14M–410M) trained for 300B tokens under three linguistically grounded curricula and Random ordering, with additional 1B runs for Random and VV, showing that curriculum gains are scale-dependent and most pronounced for capacity-constrained models. 2 Related Work We build on three research threads: curriculum learning for language model pretraining, analyses of training dynamics, and scaling effects. Curriculum learning orders examples to shape optimization, classically in easy-to-hard form ( Bengio et al. 2009 ; Elman 1993 ) . While many curriculum formulations and schedules have been proposed ( Cornacchia & Mossel 2023 ; Kong et al. 2021 ; Zhou & Bilmes 2018 ) , their implications are less clear in single-pass LLM pretraining. Empirical findings are mixed: linguistically motivated orderings can show diminishing returns at scale ( Campos 2021 ) , whereas proxy-learnability orderings can improve perplexity and downstream accuracy for larger models ( Fan & Jaggi 2023 ) ; length-based curricula may affect stability with uncertain effects on final quality ( Li et al. 2022 ; Nagatsuka et al. 2023 ) . To capture within-training changes that aggregate metrics obscure ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) , we model training trajectories with Hidden Markov Models. Prior approaches analyze trajectories via data attribution ( Koh & Liang 2017 ; Pruthi et al. 2020 ) or simulate sequence effects on metric evolution under different data orderings ( Guu et al. 2023 ; Chai et al. 2024 ) . HMMs have been used to identify consistent learning phases across random seeds ( van der Wal et al. 2025 ) ; we extend this by fitting a joint HMM across orderings to test whether curricula change phase structure or primarily reallocate exposure within shared phases. To connect ordering to mechanisms, we focus on stability diagnostics and late-stage capacity limits. Gradient noise scale provides

Read the complete text