Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer
Yihe Dong, Lorenzo Noci, Mikhail Khodak, Mufan Li · 2025
Read Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer by Yihe Dong, Lorenzo Noci, Mikhail Khodak, Mufan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01115.txt.
Introduction
Yihe Dong and co-authors ask whether random attention is enough for sequence modeling in 2025, under the title given in the text head. Transformers are the base of most large language models. Their core part is self-attention, where each word looks at others. The team tests what happens when parts are frozen at their start values.
Frozen key and query weights still let the model form key circuits. Scores on language tasks stayed strong. The team proves a new result on what such models can express. A variant with fully random attention scores also worked, past a set depth. Their reading is that attention drives in-context thought. Other layers store facts, with help from attention.
Catalog record
| CREATOR | Yihe Dong, Lorenzo Noci, Mikhail Khodak, Mufan Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01115
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer Yihe Dong, Lorenzo Noci, Mikhail Khodak, Mufan Li Abstract The Transformer architecture is central to the success of modern Large Language Models (LLMs), in part due to its surprising ability to perform a wide range of algorithmic tasks – including mathematical reasoning, memorization, and retrieval – using only gradient-based training on next-token prediction. While the core component of a Transformer is the self-attention mechanism, we question how much, and which aspects, of the performance gains can be attributed to it. To this end, we compare standard Transformers to variants in which either the multi-layer perceptron (MLP) layers or the attention projectors (queries and keys) are frozen at initialization. To further isolate the contribution of attention, we introduce MixiT – the Mixing Transformer – a simplified, principled model in which the attention scores are entirely random and fixed at initialization, eliminating any input-dependent computation or learning in attention. Surprisingly, we find that MixiT matches the performance of fully trained Transformers on various algorithmic tasks, especially those involving basic arithmetic or focusing heavily on memorization. For retrieval-based tasks, we observe that having input-dependent attention scores is consistently beneficial, while MixiT underperforms. We attribute this failure to its inability to form specialized circuits such as induction heads – a specific circuit known to be crucial for learning and exploiting repeating patterns in input sequences. Even more interestingly, we find that attention with frozen key and query projectors is not only able to form induction heads, but can also perform competitively on language modeling. Our results underscore the importance of architectural heterogeneity, where distinct components contribute complementary inductive biases crucial for solving different classes of tasks. 1 1 1 Our code is publicly available at https://github.com/princeton-pli/MixiT . 1 Introduction Transformers [ 50 ] have rapidly become the workhorse architecture in modern machine - learning systems, powering state - of - the - art models in language, vision, and scientific domains [ 15 , 48 , 22 ] . Their success is typically attributed to the self - attention mechanism, which allows every token to aggregate information from the entire sequence and has been linked to emergent abilities such as long - range retrieval, algorithmic reasoning, and in - context learning. Yet we lack a precise answer to a fundamental question: which degrees of freedom inside the Transformer are truly necessary for these behaviours, and which can be simplified away without harming performance? Prior work has probed the internals of trained Transformers. Studies of attention maps consistently report the emergence of induction heads that copy information forward and enable in - context retrieval [ 37 ] . Another line of work has focused on whether and how Transformers’ capabilities can emerge [ 1 , 41 , 24 ] by studying synthetic datasets with Zhong and Andreas 2024 showing that models with frozen self - attention but trainable embeddings still solve many algorithmic tasks. These results hint that different parts of the architecture are responsible for modeling different tasks. A standard Transformer block, however, contains several interacting components: an attention block, with learnable and input-dependent queries, keys form the attention weights, and MLP blocks composed of fully connected layers. To exemplify the complexity of the interaction, notice that the attention weights can change their value both through changes in the queries/keys parameters, as well as as through the residual - stream representations that feed those projections. Hence, because all the components are trained together, it is hard to discern the contribution of each individual one to the solution of a given task. In this work, we study the role of each Transformer’s component by freezing different parts of the architecture to their value at initialization. In partiucular, we consider the following simplified variants: • Frozen - QK , preserves the conventional attention structure but freezes the query and key weight matrices, allowing only the value weight matrix to be learned. • Frozen-MLP , where the weight matrices of the MLP block are frozen to their initial value. • MixiT (Mixing Transformer), a Transformer variant in which the attention sub - layer is replaced by a fixed, randomly initialised mixing matrix. After initialisation the mixing weights are frozen, preventing the model from adapting interactions during training or conditioning them on input content. Learning therefore takes place exclusively in the embedding and MLP blocks and in residual branches. Figure 1 : Variants of the Llama Transformer model that we study. Comparing these three simplified models with a fully - trainable Transformer makes it possible to separate the role of (i) learned token - dependent weights and (ii) the mere presence of a mixing operation, as well as (iii) understand the role of MLPs. We compare across several categories of tasks: mathematical reasoning, sentiment classification, memorization, information retrieval and language modeling. This way, we can inspect whether and how the different parts of the architecture are able to solve primitive tasks that relate to a range of basic reasoning and memorization skills. Here we summarize our main contributions: • We present the surprising finding that Frozen-QK can perform competitively with the standard transformer on language modeling tasks. Indeed, Frozen-QK develops the ability to form specialized circuits such as induction heads during training. • We find that, in all but induction heads tasks and language modeling, MixiT achieves performance comparable to the fully trained transformer and Frozen-QK. These results suggest that for a wide range of tasks, "attention mixing is all you need" regardless of its specific form, in the sense that learned query - key interactions are not required. • When tested on tasks that require forming induction heads, MixiT fails to achieve good performance, while both the fully trained transformer and Frozen-QK can solve them to close-to perfect accuracy. • On pure memorization tasks, Frozen-MLP loses two-thirds of the storage capacity of the fully-trainable Transformer, while Frozen-QK and MixiT only a third. This suggests that the MLP layers are largely responsible, but collaborates with attention, on memorization tasks. Our results indicate that the Transformer architecture has some built-in inductive bias towards algorithmic abilities, namely the ability to form circuits. Furthermore, while for a variety of tasks no particular structure in the attention maps is needed, some learnable attention mechanism is required when flexibly copying tokens. As a consequence, due to the observed gap in language modeling, we speculate that the ability to solve associative recall and complex retrieval is a key ingredient to modern language models. More broadly, our results underpin the importance of having heterogeneous components that are more specialized in efficiently solving modular subtasks, enabling the model to express complex, wide-ranging behaviors. 2 Models Specification We consider a decoder-only Transformer based on the widely used Llama architecture [ 49 ] . Given an input sequence x\in\mathbb{R}^{V\times m} , where V is the vocabulary size and m is the sequence length, embedded with a linear map to the hidden states h_{0}=W_{emb}\,x where W_{emb}\in\mathbb{R}^{V\times n} , where n is the width of the model. At its core, the transformer uses L stacked modules alternating the causal multi-head self-attention layers and MLP layers. Each self attention head is defined as: \text{Attn}(h_{\ell})=W^{v}_{\ell}h_{\ell}\,\text{Softmax}\left(\frac{1}{\sqrt{n_{h}}}Q_{\ell}^{T}K_{\ell}\right)\,,\quad Q_{\ell}=W_{\ell}^{Q}h_{\ell}\,,\quad K_{\ell}=W_{\ell}^{K}h_{\ell}\,, (2.1) where W^{Q}_{\ell},W^{K}_{\ell},W^{v}_{\ell}\in\mathbb{R}^{n_{h}\times n} are the