Learning In-context 𝒏 \boldsymbol{n} -grams with Transformers: Sub- 𝒏 \boldsymbol{n} -grams Are Near-stationary Points
Aditya Varre, Gizem Yüce, Nicolas Flammarion · 2025
Read Learning In-context 𝒏 \boldsymbol{n} -grams with Transformers: Sub- 𝒏 \boldsymbol{n} -grams Are Near-stationary Points by Aditya Varre, Gizem Yüce, Nicolas Flammarion (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12837.txt.
Introduction
Aditya Varre, Gizem Yüce, and Nicolas Flammarion explain why transformer training on language-like tasks proceeds in sudden stages, in a two thousand twenty-five machine learning paper. Long plateaus followed by abrupt progress are a familiar sight in training curves.
The authors study in-context learning of n-gram models under cross-entropy loss. They give a condition for parameter settings to be stationary points, then construct configurations implementing every smaller k-gram estimator. In the limit of long sequences and large parameter norm, the gradient vanishes at those configurations. Sub-n-grams are thus near-stationary points that training lingers near before jumping onward. Numerical experiments display the predicted stage-wise transitions.
Catalog record
| CREATOR | Aditya Varre, Gizem Yüce, Nicolas Flammarion |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 152 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-12837
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning In-context 𝒏 \boldsymbol{n} -grams with Transformers: Sub- 𝒏 \boldsymbol{n} -grams Are Near-stationary Points Aditya Varre, Gizem Yüce, Nicolas Flammarion Abstract Motivated by empirical observations of prolonged plateaus and stage-wise progression during training, we investigate the loss landscape of transformer models trained on in-context next-token prediction tasks. In particular, we focus on learning in-context n n -gram language models under cross-entropy loss, and establish a sufficient condition for parameter configurations to be stationary points. We then construct a set of parameter configurations for a simplified transformer model that represent k k -gram estimators (for k ⩽ n k\leqslant n ), and show that the gradient of the population loss at these solutions vanishes in the limit of infinite sequence length and parameter norm. This reveals a key property of the loss landscape: sub- n n -grams are near-stationary points of the population cross-entropy loss, offering theoretical insight into widely observed phenomena such as stage-wise learning dynamics and emergent phase transitions. These insights are further supported by numerical experiments that illustrate the learning dynamics of n n -grams, characterized by discrete transitions between near-stationary solutions. † † affiliationnotice: Equal contribution \declaretheorem [name=Theorem,numberwithin=section]thm-rest \declaretheorem [name=Lemma,numberwithin=section, sibling=thm-rest]lem-rest \declaretheorem [name=Corollary,numberwithin=section, sibling=thm-rest]cor-rest \declaretheorem [name=Definition,numberwithin=section, sibling=thm-rest]def-rest \declaretheorem [name=Proposition,numberwithin=section, sibling=thm-rest]prop-rest \declaretheorem [name=Assumption,numberwithin=section, sibling=thm-rest]ass-rest 1 Introduction Transformers ( Vaswani et al. 2017 ) have become central to modern machine learning, due to their capabilities such as in-context learning (ICL) ( Brown 2020 ) —the ability of models to perform new tasks by leveraging a few examples provided within the context, without the need for parameter updates or retraining. Recent empirical studies have revealed that the dynamics that result in in-context learning abilities often deviate from a simple monotonic decrease in loss, exhibiting complex behaviors with plateaus such as grokking ( Power et al. 2022 ) and stage-wise transitions ( Olsson et al. 2022 ) where the training process frequently lingers in regions of slow progress before going through a sudden phase transition after which ICL abilities are acquired. Mechanistic interpretability studies suggest that after these plateaus, specific circuits, such as induction heads ( Olsson et al. 2022 ) , or syntactic structures ( Chen et al. 2024a ) , are gradually learned. This raises a natural question: Why does training linger at plateaus before developing such abilities? The setting of Edelman et al. 2024 provides an avenue to take a closer look at this question with the specialized task of learning in-context n -grams ( Shannon 1948 ; Chomsky 1956 ; Brown et al. 1992 ) . Here, the training unfolds in a hierarchical, phase-wise manner. It begins with the transformer making uniform predictions, progresses through unigram and bigram predictions, and potentially generalizes to higher-order n -grams. Figure 1 illustrates how these training phases overlap with the losses of the sub- n -gram estimators. Building on this observation, we provide a theoretical foundation for why training lingers at long plateaus—the loss landscape of transformers trained on in-context sequential data exhibit stationary points aligned with sub-hierarchical or sub-syntactic solutions. These points act as intermediate solutions where gradients vanish, causing training to stagnate before transitioning to the next hierarchical level. Concretely, this paper explores the loss landscape of transformer models trained on in-context n -gram language models to predict the next token. We show a sufficient stationarity condition for the cross-entropy loss that the sub- n -gram constructions satisfy, shedding light on the incremental and phase-wise learning phenomena observed during training for in-context n -grams. Our main contributions can be summarized as follows: • In Section 3 , we provide a sufficient condition for the solutions to be the stationary points of the cross-entropy loss in the next-token prediction task when the model’s score function depends solely on a sub-sequence of the input history. This characterization provides a powerful tool for analyzing the structure of the loss landscape for various tasks, including but not limited to n -grams. • In Section 4 , we demonstrate that a set of solutions representing sub- n -gram estimators are near-stationary points, by verifying that they satisfy the conditions sufficient for the gradient of the population loss to converge to zero in the infinite weight and sequence length limit. • In Section 5 , we present empirical evidence that illustrates the structural evolution of these models during training and how transitions between training phases are in alignment with the predictions of our theory. Figure 1 : The stage-wise behavior of the test loss during training for the 4 -gram language model. The dashed lines represent the cross entropy loss of the k -gram estimators for k={1,2,3,4} . The plateaus of the test loss overlap with the losses of k -gram estimators. 1.1 Related Work In-context Learning. To understand in-context learning (ICL) ( Brown 2020 ) , previous works have explored various approaches. One approach is mechanistic interpretability, which has revealed the emergence of circuits called induction heads during ICL ( Olsson et al. 2022 ) . Another approach involves studying ICL on specific hypothesis classes to understand how transformers solve these tasks in context. A common feature of the training dynamics in these studies is the presence of plateaus ( Chen et al. 2024a ; Kim et al. 2024 ) , after which the models acquire certain capabilities. Examples include studying ICL on regression tasks ( Garg et al. 2022 ; Von Oswald et al. 2023 ; Ahn et al. 2024 ) , boolean functions ( Bhattamishra et al. 2023 ) , regular languages ( Akyürek et al. 2024 ) , and n -grams. (In-context) Learning \bm{n} -gram language models. n -gram language models, or higher-order Markov chains, are effective mathematical models for generating sequential data and capturing certain aspects of natural language ( Shannon 1948 ; Jelinek 1998 ; Jurafsky & Martin 2009 ) . As a result, many studies have focused on analyzing transformers through the lens of Markov sequences. Svete & Cotterell 2024 examines the representational limits of transformers on n -grams, while Rajaraman et al. 2024 focuses on the in-context counterpart. Makkuva et al. 2024 explores the landscape of transformers on data from binary first-order Markov chains (bigrams) but does not consider the in-context setting. Bietti et al. 2024 investigates the formation of induction heads needed to learn bigrams with specific trigger tokens. Additionally, Nichani et al. 2024 studies the formation of induction heads through gradient descent in learning causal structures. A closely related work to ours is Edelman et al. 2024 . They report the stage-wise dynamics of transformers while learning in-context n -grams, and identify that these stages correspond to sub- n -grams. However, their theoretical analysis is limited to one step of gradient descent on binary bigrams, while we characterize the sub- n -grams as near-stationary points for unrestricted n and vocabulary size, explaining the long plateaus once the training reaches a state expressing these solutions. Chen et al. 2024b study the same in-context n -gram prediction task, but with an architecture that involves a feed-forward network layer that does the token selection. However, they have a threee-stage training procedure and an initialization scheme that ensures different heads attend to different tokens from the start, eliminating the stage-wise learning dynamics we study. Sequential learning. Fukumizu & Amari