On the Emergence of Induction Heads for In-Context Learning

Tiberiu Mușat, Tiago Pimentel, Lorenzo Noci, Thomas Hofmann · 2025

Read On the Emergence of Induction Heads for In-Context Learning by Tiberiu Mușat, Tiago Pimentel, Lorenzo Noci, Thomas Hofmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01033.txt.

Introduction

Tiberiu Musat and colleagues investigate induction heads, a circuit pattern inside transformer models that helps them learn from context. Induction heads copy and complete patterns seen earlier in the input, and they are thought to be a key mechanism behind in context learning.

Working with a minimal task and a modified two layer transformer, the team finds the weight matrices that implement an induction head have a simple, interpretable structure. They prove training dynamics stay inside a nineteen dimensional subspace of the parameter space, and in practice only three dimensions account for the head's emergence. Inside that small subspace, the time until the head appears grows quadratically with context length, a tight bound they establish.

Catalog record

CREATORTiberiu Mușat, Tiago Pimentel, Lorenzo Noci, Thomas Hofmann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01033

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Emergence of Induction Heads for In-Context Learning Tiberiu Mușat, Tiago Pimentel, Lorenzo Noci, Thomas Hofmann Abstract Transformers have become the dominant architecture for natural language processing. Part of their success is owed to a remarkable capability known as in-context learning (ICL): they can acquire and apply novel associations solely from their input context, without any updates to their weights. In this work, we study the emergence of induction heads , a previously identified mechanism in two-layer transformers that is particularly important for in-context learning. We uncover a relatively simple and interpretable structure of the weight matrices implementing the induction head. We theoretically explain the origin of this structure using a minimal ICL task formulation and a modified transformer architecture. We give a formal proof that the training dynamics remain constrained to a 19-dimensional subspace of the parameter space. Empirically, we validate this constraint while observing that only 3 dimensions account for the emergence of an induction head. By further studying the training dynamics inside this 3-dimensional subspace, we find that the time until the emergence of an induction head follows a tight asymptotic bound that is quadratic in the input context length. 1 Introduction How does intelligence emerge from gradient descent ? Large language models (LLMs) have achieved highly advanced reasoning abilities, yet we still lack a principled account of how complex reasoning behaviors emerge from this simple learning rule. Understanding the inner workings of LLMs is an important avenue towards developing novel AI systems with increased reliability and efficiency. LLMs possess a remarkable ability known as in-context learning (ICL). A well-trained language model can learn and apply novel associations from their input context, without additional parameter updates ( Brown et al. 2020 ) . This is in stark contrast to traditional in-weights learning , where novel associations are directly encoded into the model weights. Previous work by Olsson et al. 2022 traces back the majority of transformers’ ICL capabilities to a learned mechanism termed induction head : a pair of two consecutive attention heads that implement a simple but powerful copying rule \left[\;\ldots,A,B,\ldots,A\;\right]\to B . Empirical work has shown that the formation of induction heads co-occurs with a sharp decrease in the training loss and an increase in ICL accuracy ( Olsson et al. 2022 ; Reddy 2023 ) . While a number of theoretical studies have established the emergence of induction heads using specific staged learning algorithms ( Nichani et al. 2024a ; Bietti et al. 2024 ) , the precise dynamics during standard training remain elusive. This motivates the question of the current study: How do induction heads emerge during training? To answer this question, we study the training dynamics of an autoregressive two-layer transformer using a minimal ICL task formulation (defined in section 3 ) and a simplified architecture. We show that in the proposed setup, only 19 dimensions of parameter space have non-zero gradients and therefore govern the entire learning trajectory. Then, we empirically show how only 3 dimensions of the parameter space are needed to form an induction head. In this reduced and interpretable parameter space, we explicitly study the dynamics of the three pseudo-parameters and analyze the formation of induction heads. Concretely, our contributions are as follows: 1. We train and interpret a standard attention-only transformer on an ICL task ( section 2 ). We find a relatively simple and highly interpretable description of the weight matrices that implement the induction head. 2. Using a minimal ICL formulation, we give a formal proof that training dynamics induce a simplified structure of the weights ( section 4 ). The evolution of model weights stays within a 19-dimensional subspace of the entire parameter space, regardless of model or task size. We index this subspace by introducing 19 pseudo-parameters . 3. We empirically find that only 3 pseudo-parameters are learned at the end of training, corresponding exactly to an induction head ( section 5 ). We also find that the emergence of the 3 parameters is self-contained , unaided by the presence of the other 16 parameters. 4. We theoretically study the training dynamics of the induction head, assuming that only the 3 parameters are learnable ( section 6 ). We prove that the 3 parameters always emerge in a specific sequence. We also prove asymptotic bounds for the emergence time for each parameter in terms of the context length, as well as a tight bound on the total emergence time . Finally, we also provide empirical validation for our theoretical results. Figure 1: Left: an induction head solving the in-context learning (ICL) task. Given a series of item-label pairs, the model predicts the correct label for a query item. The first attention head retrieves the corresponding item for each label, enabling the second attention head to retrieve the correct label. Each path is modulated by one pseudo-parameter ( \alpha_{3} , \beta_{2} , or \gamma_{3} ). Right: our minimal transformer architecture. We use two attention-only layers and a linear layer. We disentangle the attention layers by concatenating the inputs and outputs, rather than adding them together. 2 Induction Heads Induction heads are attention heads that implement a simple but powerful algorithm. Given a prompt of the form \left[\;\ldots,A,B,\ldots,A\;\right] , an induction head predicts the token which follows the previous occurrence of A , in this case being B . Note that induction heads are not a modified type of attention head, but rather a mechanism learned by regular attention heads during standard training. Induction heads are composed of two attention layers. The first attention layer retrieves the value of A into B by attending to the previous token using positional embeddings. The newly obtained value enables the second attention layer to retrieve B from the second occurrence of A . Note that two layers are necessary to solve the task since B and the second A initially have no shared information. 2.1 Setup In order to understand how induction heads are implemented, we train an autoregressive transformer following the recipe of Vaswani et al. 2017 . We train the model using synthetic data to predict the label of a query item based on the preceding item-label pairs, as depicted in fig. 1 (left). We use only two attention-only layers with one attention head per layer. We remove MLPs from the transformer blocks since they are neither necessary nor useful for the task at hand. We apply a cross-entropy loss only on the final (the query item). We specify the full training details in appendix E . 2.2 Weight Matrix Structure Notation. We train a transformer with residual stream dimension D , vocabulary size N_{T} , and block size N_{P} . We denote the embedding of token i as {\bm{t}}_{i}\in\mathbb{R}^{D} and the embedding of position j as {\bm{p}}_{j}\in\mathbb{R}^{D} . We deonte the token and positional embedding matrices as {\bm{T}}\in\mathbb{R}^{D\times N_{T}} and {\bm{P}}\in\mathbb{R}^{D\times N_{P}} , respectively. The layer l\in\{1,2\} has the query, key, value, and output matrices {\bm{W}}_{Q}^{l},{\bm{W}}_{K}^{l},{\bm{W}}_{V}^{l}\in\mathbb{R}^{D_{H}\times D} , and {\bm{W}}_{O}^{l}\in\mathbb{R}^{D\times D_{H}} , respectively. A final linear output layer {\bm{W}}_{o}\in\mathbb{R}^{N_{T}\times D} is applied. We use N_{T}=N_{P}=32 . Architecture. For an input sequence of length L with tokens {\bm{X}}_{T}\in\mathbb{R}^{N_{T}\times L} and positions {\bm{X}}_{P}\in\mathbb{R}^{N_{P}\times L} encoded as one-hot vectors, the model will compute residual streams {\bm{H}}_{l}\in\mathbb{R}^{D\times L}

Read the complete text