Rethinking Associative Memory Mechanism in Induction Head

Shuo Wang, Issei Sato · 2024

Read Rethinking Associative Memory Mechanism in Induction Head by Shuo Wang, Issei Sato (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11459.txt.

Introduction

Shuo Wang and Issei Sato wrote a 2024 paper on how large language models learn from examples placed inside a prompt. They focus on the induction head, a part of the model linked to this in-context learning. Their question is how the model mixes fresh prompt facts with word links it learned during training.

Working with a two-layer transformer, the authors study the weights in the attention layers as a form of linked memory. They work out what the model should output when prompts come from a simple word-pair model. They then build test prompts to check whether a trained model matches the theory. The study helps explain how models balance new context against stored knowledge.

Catalog record

CREATORShuo Wang, Issei Sato
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE137 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11459

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Associative Memory Mechanism in Induction Head Shuo Wang, Issei Sato Abstract Induction head mechanism is a part of the computational circuits for in-context learning (ICL) that enable large language models (LLMs) to adapt to new tasks without fine-tuning. Most existing work explains the training dynamics behind acquiring such a powerful mechanism. However, the model’s ability to coordinate in-context information over long contexts and global knowledge acquired during pretraining remains poorly understood. This paper investigates how a two-layer transformer thoroughly captures in-context information and balances it with pretrained bigram knowledge in next token prediction, from the viewpoint of associative memory. We theoretically analyze the representation of weight matrices in attention layers and the resulting logits when a transformer is given prompts generated by a bigram model. In the experiments, we design specific prompts to evaluate whether the outputs of the trained transformer align with the theoretical results. 1 Introduction In recent years, transformer-based models, such as BERT [ 14 ] and GPT [ 44 , 45 , 6 ] , have achieved remarkable success in natural language processing. Especially, in-context learning (ICL) [ 6 , 15 ] has emerged as a groundbreaking capability within large language models (LLMs), enabling them to adapt to new tasks without traditional fine-tuning. Instead, these models leverage patterns from a prompt or input sequence, effectively learning "in context" by interpreting examples or instructions provided in real-time [ 35 , 58 ] . During a phase change, models acquire the ability to complete complex patterns through induction heads, suggesting that induction heads are a key mechanistic basis of ICL [ 42 ] . It is known that the induction head mechanism can emerge in two-layer transformers [ 17 ] , and many theoretical studies have focused on analyzing two-layer architectures to understand how induction head develops [ 16 , 41 , 10 ] . Among them, Bietti et al. [5] demonstrated, both theoretically and empirically, that training a transformer leads to weight matrices behaving as associative memories, composing the induction head mechanism. However, the effectiveness of the induction head mechanism tends to diminish for the latter part of a sequence [ 5 ] , which does not align with the behavior observed in real-world LLM. In addition, while the learning dynamics of the induction head have been elucidated in the existing work, it remains unclear how in-context knowledge provided in the prompt and global knowledge acquired during pretraining influence the output of two-layer transformers. Understanding how a transformer predicts the next token is crucial, as it allows us to model phenomena such as context hijacking [ 27 ] , in which altering the context disrupts factual recall, causing the model to produce incorrect outputs influenced by misleading in-context information. In this paper, we theoretically and empirically show, from the perspective of associative memory, how a two-layer transformer with relative positional encoding (RPE) can avoid failure in detecting patterns that appear in the latter part of a sequence. We also investigate how the model prioritizes in-context knowledge versus global knowledge when predicting the next token, using training data generated by a bigram model with triggered transitions. To summarize, we make the following contributions: • We theoretically show that the induction head learned by a transformer using RPE can avoid oversight of patterns within the framework of associative memory. • We experimentally confirmed that the model with RPE can comprehensively use in-context knowledge for the bigram model. We also demonstrated its ability to effectively infer on sequences where the transformer using absolute positional encoding (APE) fails in next-token prediction. • We investigated how the model prioritizes in-context knowledge versus global knowledge when generating outputs, using both theoretical analysis and experimental evaluation. 2 Related work Paper Model Objective Theoretical analysis Meng et al. [37] GPT knowledge editing no Cabannes et al. [7] linear model scaling laws yes Cabannes et al. [8] linear model learning dynamics yes Jiang et al. [27] one-layer transformer concept association yes Chen et al. [9] linear model noisy classification yes Bietti et al. [5] two-layer transformer learning dynamics yes OURS two-layer transformer length generalization & global v.s. in-context yes Table 1 : Comparisons among related work and ours in terms of associative memory. The concept of associative memory is widely adopted for theoretical analyses of various phenomena. Associative memory Associative memory refers to the storage and retrieval of patterns, and its computational model was designed to retrieve complete memories from sufficient fragments of information in neuroscience [ 3 , 22 , 23 ] . After the emergence of modern Hopfield networks [ 32 ] , which store patterns in an energy function and retrieve them using an updating rule. Moreover, many studies employ the concept of associative memory in various ways. For instance, Zhang et al. [60] propose a memory unit that estimates a conditional probability distribution with Gaussian kernel smoothing. Some work considers attention blocks as memory because they read and write information flowing through the residual stream [ 41 , 18 ] . Meng et al. [37] locate factual association patterns in the weight matrix of an MLP. Recently, it has become increasingly popular to regard a matrix as associative memory from a theoretical perspective, as shown in Tab. 1 . Previous studies examine scaling laws in relation to model capacity [ 7 ] , learning dynamics of linear model in terms of particle system [ 8 ] , and the effect of low-rank approximation of weight matrix to classification involving a noisy token [ 9 ] for a linear model. Additionally, Jiang et al. [27] demonstrate that a one-layer transformer can recover latent concept from long enough context. In-context learning and induction head Since Brown [6] introduced ICL, theoretical understanding of this phenomenon has been a major interest within the community. For example, Akyürek et al. [2] showed that transformer can implement a learning algorithm that solves linear regression. Von Oswald et al. [53] demonstrated that linear self-attention layer can emulate gradient descent. Similarly, many studies [ 1 , 36 , 61 , 12 ] examined the theoretical connections between ICL and gradient descent. Mechanistic interpretability, which aims to understand the model behavior by reverse-engineering its internal components [ 39 , 55 , 11 ] , is another perspective for analyzing ICL [ 42 , 17 ] . In particular, Elhage et al. [17] discovered that two-layer attention-only transformers understand the previous use of the token and look for similar situations in the context. In addition, Olsson et al. [42] provided empirical evidence that induction heads are the mechanism behind ICL. Moreover, Bansal et al. [4] identified that many induction heads are also among the attention heads deemed critical for ICL. The most relevant study to ours is by Bietti et al. [5] , who investigated the learning dynamics of the induction head mechanism in two-layer transformers from the perspective of associative memory. In contrast, our study focuses on the functionality of the induction head mechanism and the influence of knowledge acquired during training and that obtained in context on the final output. 3 Preliminaries 3.1 Transformer architecture In this work, we focus on a two-layer transformer described below. Calculation Given a sequence of tokens z_{1:T} of length T\in\mathbb{N} from the vocabulary

Read the complete text