Predicting the Formation of Induction Heads

Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan Schneider · 2025

Read Predicting the Formation of Induction Heads by Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan Schneider (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16893.txt.

Introduction

Tatsuya Aoyama, Ethan Gotlieb Wilcox, and Nathan Schneider predict when induction heads, the attention circuits credited with language models' in-context learning, will appear during pre-training. Despite their importance, the exact conditions that trigger these pattern-copying heads in language modeling had not been pinned down.

A simple equation combining batch size and context size forecasts the emergence point, and that point barely depends on model size. The statistical fuel is bigram repetition: how often token pairs repeat and how reliably they predict what follows jointly set an effective decision boundary for head formation. Local dependencies with frequent, reliable repetition suffice to create induction heads, while category structure and the shape of the token distribution modulate behavior near the boundary. The findings link a core learning mechanism to measurable data statistics.

Catalog record

CREATORTatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan Schneider
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-16893

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Predicting the Formation of Induction Heads Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan Schneider Abstract Arguably, specialized attention heads dubbed induction heads (IHs) underlie the remarkable in-context learning (ICL) capabilities of modern language models (LMs) ; yet, a precise characterization of their formation remains unclear. In this study, we investigate the relationship between statistical properties of training data (for both natural and synthetic data) and IH formation. We show that (1) a simple equation combining batch size and context size predicts the point at which IHs form; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs , and we find a precise Pareto frontier in terms of these two values; and (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but when the frequency and reliability are low, categoriality and the shape of the marginal distribution matter. \workshoptitle First Workshop on CogInterp: Interpreting Cognition in Deep Learning Models 1 Introduction The practical utility of modern LMs depends heavily on their ability to perform ICL , broadly construed as performing a task based on the input provided at inference time. Various accounts have been provided to explain the internal workings of this capability, and among them are studies that find certain attention heads affecting LMs ’ ICL capabilities [ 11 , 6 , 15 ] . The emergence of such heads during pretraining is sometimes referred to as phase change or phase transition [ 4 , 1 ] . Factors that affect phase transition, particularly the formation of IHs , which are characterized by their (often long-range) in-context copying behavior, are not fully understood. 3 study the data properties that lead to different learning outcomes in a few-shot learning of image classification, where few-shot examples are image-label pairs. They find that (1) burstiness (similar things appearing in clusters), (2) within-class variation, and (3) dynamic class membership all promote ICL and demote in-weight learning (IWL) . Interestingly, matching the marginal distribution of the labels to a Zipfian distribution was the only variable that led to high ICL and IWL simultaneously. 6 proposes a synthetic task dubbed ICL Markov Chain ( ICL -MC) using a Markov Process involving 2–8 symbols. Taking a Bayesian approach to ICL [ 14 ] , 6 randomly initialize the transition matrix at the beginning of each epoch, thereby making it impossible to learn the underlying distribution of the symbols. They find that the models go through phases where their predictions most closely match the uniform distribution, then in-context unigram counts, and lastly in-context bigram counts. They show that by forming IHs that attend to all of the bigram continuations of the current token in the context, models achieve the Bayes-optimal bigram solution. In the current literature including the aforementioned studies, ICL is often studied as a few-shot learning (e.g., 3 ) with input-output pairs, or with an unrealistic setting that necessitates ICL as opposed to IWL (e.g., 6 ). As such, this raises the question of why LMs trained on natural language form IHs (and ICL capabilities). Furthermore, the point at which IH form is not fully understood. Some report a narrow range (1B–3B pretraining tokens; e.g., 11 ) while others find a much wider range (64M–2B pretraining tokens; 1 ). To fill these gaps, in this study, we analyze how pretraining configuration and data properties affect whether IHs emerge and at what point they first appear . Through a series of experiments using both natural and synthetic data, we show that (1) a simple equation combining batch size and context size can predict the IH formation point ( Section 3 ); (2) the surface bigram repetition frequency and reliability strongly affect the formation of IHs , and a precise Pareto frontier can be described in terms of these two values ( Section 4 ); and (3) local dependency (non-independence between two consecutive tokens) with high frequency and reliability are sufficient for IH formation, but when the frequency and reliability are low, the presence of latent categories of symbols and the shape of the marginal distribution matter ( Section 5 ). 2 Methods 2.1 Metrics IHs are defined by the copying behavior, such that if the model has seen an \ab AB sequence in-context and the current token is \tok A, then an IH is a head that promotes the prediction of \tok B as the next token, completing the \abab AB sequence. Following 11 , we quantify this behavior using prefix-matching score (PS) . Given a random sequence of tokens \mathbf{x} repeated twice, PS of a head h at layer l is its average attention from the source token x_{i} to the next token of its previous occurrence: \frac{1}{|\mathbf{x}|-1}\sum_{i=|\mathbf{x}|+1}^{2|\mathbf{x}|}\mathrm{\alpha}^{(h,l)}(x_{i},x_{i-(|\mathbf{x}|-1)}) . As to the size of the sequence \mathbf{x} , TransformerLens library [ 9 ] adopts |\mathbf{x}|=50 , which we also do; however, when analyzing models with smaller context sizes, we adjust |\mathbf{x}| accordingly: |\mathbf{x}|=\min(\frac{\text{|context|}}{2},50) . 2.2 Models and Checkpoints We use a toy version of GPT2 [ 12 ] with 2 layers and 8 attention heads per layer for all experiments. For experiments with natural language, we adopt a pretrained GPT2 tokenizer with a vocabulary size of 50,257, unless otherwise specified. All models analyzed in this study were trained from scratch for 1B pretraining tokens. We save intermediate checkpoints at 250K and 500K tokens, [1M, 10M) tokens at 1M increments, [10M, 100M) tokens at 10M increments, and [100M, 1B] tokens at 100M increments, resulting in 30 checkpoints per model. We additionally use pretrained Pythia models [ 2 ] for a follow-up analysis (e.g., Appendix D ). 2.3 Data For natural texts ( Section 3 ), unless otherwise specified, we use the English subcorpus from the Common Crawl Corpus ( CC100 ; 5 , 13 ). We create a sample of 100M tokens from this corpus, using the pretrained GPT2 tokenizer. As mentioned earlier, all models are trained for 10 epochs on this sample, for the total of 1B tokens. For semi-synthetic texts ( Section 4 ), we use a token-to-token transition matrix obtained from the same sample from the CC100 corpus. 3 Experiment 1: Natural Data 3.1 Methods As briefly introduced earlier, 1 find that training an LM with different batch sizes results in different phase transition points. In this experiment, we change context size and batch size to investigate their effect on the formation of IHs . Specifically, we experiment with log-spaced batch and context sizes of \{4,8,16,32,64,128,256,512\} , and \{4,8,16,32,64,128,256,512,1024,2048\} , respectively. Since grid search is expensive, we fix the batch size at 16 while changing the context size, and fix the context size at 1024 while changing the batch size. We additionally test the effect of bigram repetitions alone by selecting subsets of the pretraining data and manipulating the proportion of bigram repetitions. \displaystyle{10^{1}} \displaystyle{10^{2}} \displaystyle{10^{3}} \displaystyle{10^{4}} \displaystyle{10^{5}} \displaystyle{10^{6}} \displaystyle{10^{7}} \displaystyle{0.0} \displaystyle{0.2} \displaystyle{0.4} \displaystyle{0.6} \displaystyle{0.8} Best PS BS 4 8 16 32 64 128 256 512 \displaystyle{10^{1}} \displaystyle{10^{2}} \displaystyle{10^{3}} \displaystyle{10^{4}} \displaystyle{10^{5}} \displaystyle{10^{6}} \displaystyle{10^{7}} # Updates \displaystyle{0.0} \displaystyle{0.2} \displaystyle{0.4} \displaystyle{0.6} \displaystyle{0.8} CS 4 8 16 32 64 128 256 512 1024 2048 \displaystyle{10^{1}} \displaystyle{10^{2}} \displaystyle{10^{3}} \displaystyle{10^{4}} \displaystyle{10^{5}} \displaystyle{10^{6}} \displaystyle{10^{7}} \displaystyle{0.1} \displaystyle{0.2} \displaystyle{0.3} %NR 30% 40% 50% 60% 70% Figure 1: Developmental trajectories of PS of LMs

Read the complete text