Residual Matrix Transformers: Scaling the Size of the Residual Stream
Brian Mak, Jeffrey Flanigan · 2025
Read Residual Matrix Transformers: Scaling the Size of the Residual Stream by Brian Mak, Jeffrey Flanigan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22696.txt.
Introduction
Brian Mak and Jeffrey Flanigan replace in a 2025 study the transformer's shared memory lane, the residual stream, with a memory matrix built from outer products. Layers write into and read from that matrix, whose size can grow without growing the model's compute in step. They call the design the Residual Matrix Transformer.
It matched the standard transformer's loss with about forty percent fewer operations, a quarter fewer parameters, and two fifths fewer training tokens, and beat it on downstream tests. Their theory credits better scaling of the memory and cleaner signal spread. The memory bus, long taken for granted, turns out to be upgradeable.
Catalog record
| CREATOR | Brian Mak, Jeffrey Flanigan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22696
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Residual Matrix Transformers: Scaling the Size of the Residual Stream Brian Mak, Jeffrey Flanigan Abstract The residual stream acts as a memory bus where transformer layers both store and access features ( Elhage et al. 2021 ) . We consider changing the mechanism for retrieving and storing information in the residual stream, and replace the residual stream of the transformer with an outer product memory matrix ( Kohonen 1972 , Anderson 1972 ). We call this model the Residual Matrix Transformer (RMT). We find that the RMT enjoys a number of attractive properties: 1) the size of the residual stream can be scaled independently of compute and model size, improving performance, 2) the RMT can achieve the same loss as the transformer with 58% fewer FLOPS, 25% fewer parameters, and 41% fewer training tokens tokens, and 3) the RMT outperforms the transformer on downstream evaluations. We theoretically analyze the transformer and the RMT, and show that the RMT allows for more efficient scaling of the residual stream, as well as improved variance propagation properties. Code for this project can be found at https://github.com/bmac3/residual-matrix-transformer . 1 Introduction Kaplan et al. 2020 presented a new way forward for the field of NLP and AI, showing that simply scaling model size, data size, and computational budget is sufficient to advance model capabilities at an unprecedented rate. Their work lead to the the creation of many of the frontier models today ( Achiam et al. 2023 , Hoffmann et al. 2024 , Dubey et al. 2024 ). Meanwhile, data, compute, and model sizes continue to grow exponentially each year ( AI 2024 ) . In fact, we are quickly scaling towards the limits on how much data and energy is available ( Muennighoff et al. 2023 , Zimmerman et al. 2023 ). This presents a need for more data and compute efficient models. Figure 1: Model size and per-example FLOPS versus residual stream size for the transformer and RMT. See § 3.1 for details. When it comes to large scale LLM training, few architectural modifications rival the impact that Mixture of Expert Models ( Fedus et al. 2022 ) have had. Their work inspired many later cutting-edge models ( Du et al. 2022 , Jiang et al. 2024 ). Their Switch Transformer model had the ability to scale along the sparse parameter axis. This meant that model size could be scaled without affecting per-example compute. With this, they were able to scale LLM training in ways that had previously been unexplored. In this new search space, they found a training configuration that was 7 times more efficient than standard transformer training. In this paper, we follow a similar formula and present a new transformer-type model, the Residual Matrix Transformer (RMT), that can scale the residual stream size while keeping model size and per-example compute fixed . The residual stream is defined as the sum of all the outputs of the previous layers and acts as a communication channel between layers ( Elhage et al. 2021 ) . The size (i.e. dimension) of the residual stream determines how many features it can store, so scaling its size can be thought of as scaling its bandwidth. In our experiments, we find that scaling the size of the residual stream while maintaining the constant model size and per-example compute improves both data and compute efficiency (§ 4.4 ). Scaling the residual stream in a standard transformer resizes all the parameter matrices, which linearly influences the parameter and FLOP counts (see Fig. 1 ). To address this, we replace the residual stream representation with an outer product memory matrix ( Kohonen 1972 , Anderson 1972 ), and find the relationship between the size of the residual stream and parameter and FLOP count becomes nearly constant for typical transformer dimensions. Additionally, we find that the our proposed model is more efficient in terms of data, FLOPS, and parameters than the standard transformer. We summarize our contributions as follows: • We introduce a new transformer-type model, the Residual Matrix Transformer (RMT), that replaces the residual stream with an outer product memory matrix (§ 2 ). • We experimentally find that the RMT is more efficient in terms of data, FLOPS, and parameters than the transformer. The RMT trains with fewer resources and achieves better performance than the transformer and variants (§ 4.2 , § 4.3 , and § 4.5 ). • We present theory that shows the RMT exhibits efficient scaling of the residual stream and has improved variance propagation properties (§ 3 ). • We experimentally explore the effects of scaling the residual stream size while keeping model size and per-example compute fixed with the RMT. We find that a larger residual stream improves performance (§ 4.4 ). 2 Model Architecture In this section we introduce the Residual Matrix Transformer, a transformer variant whose residual stream is replaced with an outer product memory mechanism. We first review the transformer architecture (§ 2.1 ), and then present the RMT formulation (§ 2.2 ). 2.1 Transformer Transformer Overview We begin by giving the equations that describe the transformer architecture ( Vaswani et al. 2017 ) . In our formulation, the input to the transformer is a matrix {\bm{S}}\in\mathbb{R}^{V\times N} whose columns are the one-hot-vectors of some tokenized string. The first layer encodes the input. \displaystyle{\bm{X}}^{(0)}={\textnormal{E}}({\bm{S}})+{\textnormal{P}}({\bm{N}}) (1) Here, {\bm{N}}\in\mathbb{R}^{N\times N} are the one-hot-vectors representing each token’s position. The output of this layer is an embedding matrix {\bm{X}}^{(0)}\in\mathbb{R}^{D\times N} , and it contains the first instance of each token’s residual stream. The next 2L layers are defined recursively and describe the successive application of attention and feed-forward layers to the residual stream. \displaystyle{\bm{X}}^{(l)}={\textnormal{MHA}}({\textnormal{LN}}({\bm{X}}^{(l-1)}))+{\bm{X}}^{(l-1)} (2) \displaystyle{\bm{X}}^{(l+1)}={\textnormal{FF}}({\textnormal{LN}}({\bm{X}}^{(l)}))+{\bm{X}}^{(l)} (3) where LN is LayerNorm ( Xu et al. 2019 ) . The final layer reads out the logits of the next predicted token from the residual stream. The transformer T is thus represented as: \displaystyle{\textnormal{T}}({\bm{S}})={\textnormal{U}}({\textnormal{LN}}({\bm{X}}^{(2L)})) (4) Embedding Layer The embedding layer encodes the one-hot-vector matrices using the learned embeddings {\bm{W}}_{E}\in\mathbb{R}^{D\times V} and {\bm{W}}_{PE}\in\mathbb{R}^{D\times N} . \displaystyle{\textnormal{E}}({\bm{S}})={\bm{W}}_{E}{\bm{S}} (5) \displaystyle{\textnormal{P}}({\bm{N}})={\bm{W}}_{PE}{\bm{N}} (6) Attention Layer For our attention layer definition, we group the computation of each attention head separately. \displaystyle{\textnormal{MHA}}({\bm{X}})=\sum_{h=1}^{H}{\bm{W}}^{(h)}_{O}{\textnormal{SHA}}({\bm{Q}}^{(h)},{\bm{K}}^{(h)},{\bm{V}}^{(h)}) (7) \displaystyle{\textnormal{SHA}}({\bm{Q}},{\bm{K}},{\bm{V}})=\textrm{Softmax}(\frac{{\bm{Q}}^{T}{\bm{K}}}{\sqrt{D_{h}}}){\bm{V}} (8) \displaystyle{\bm{Q}}^{(h)}={\bm{W}}^{(h)}_{Q}{\bm{X}}\quad{\bm{K}}^{(h)}={\bm{W}}^{(h)}_{K}{\bm{X}}\quad{\bm{V}}^{(h)}={\bm{W}}^{(h)}_{V}{\bm{X}} (9) {\bm{W}}^{(h)}_{Q},{\bm{W}}^{(h)}_{K},{\bm{W}}^{(h)}_{V}\in\mathbb{R}^{D_{h}\times D} and {\bm{W}}^{(h)}_{O}\in\mathbb{R}^{D\times D_{h}} . FeedForward Layer We define our feed-forward network in the standard way, using a Gelu activation. {\bm{W}}_{1}\in\mathbb{R}^{D_{FF}\times D} and {\bm{W}}_{2}\in\mathbb{R}^{D\times D_{FF}} . \displaystyle{\textnormal{FF}}({\bm{X}})={\bm{W}}_{2}{\textnormal{Gelu}}({\bm{W}}_{1}{\bm{X}}) (10) Unembedding Layer The unembedding layer multiplies the final instance of the residual stream with unembedding weights {\bm{W}}_{U}\in\mathbb{R}^{V\times D} . \displaystyle{\textnormal{U}}({\bm{X}})={\bm{W}}_{U}{\bm{X}} (11) 2.2 Residual Matrix Transformer Here we give relevant technical background on outer product memories and show how this mechanism can be applied to the transformer. Outer Product Memories We propose to replace the residual stream with an outer product memory store. An outer product memory store is created by summing the outer products of a set of key-value vector pairs ( Kohonen 1972 , Anderson 1972 , Gmitro et al. 1989 ). For some set of key vectors \{{\bm{q}}^{(p)}\in\mathbb{R}^{D_{k}}\}^{N}_{p=1} and data vectors \{{\bm{x}}^{(p)}\in\mathbb{R}^{D_{v}}\}^{N}_{p=1} , an outer product store {\bm{M}}\in\mathbb{R}^{D_{k}\times D_{v}} can be constructed using the following equation: \displaystyle{\bm{M}}=\textnormal{Norm}(\sum^{N}_{p=1}{\bm{q}}^{(p)}\otimes{\bm{x}}^{(p)}) (12) Here {\bm{u}}\otimes{\bm{v}}={\bm{u}}{\bm{v}}^{T} . Norm is a normalization function, which in the case of RMT is LayerNorm. One can retrieve a particular data vector {\bm{x}}^{(r)} using its corresponding