Gradient Boosting within a Single Attention Layer

Saleh Sargolzaei · 2026

Read Gradient Boosting within a Single Attention Layer by Saleh Sargolzaei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03190.txt.

Introduction

Sargolzaei adds a second attention pass inside a single layer so the layer can correct its own errors, borrowing the logic of gradient boosting. The correction pass watches the first pass's error and applies a gated fix, improving perplexity by about 6 percent on two text benchmarks. The trick needs the additive residual layout of Pre-LN transformers; under Post-LN, the same design degrades perplexity.

Catalog record

CREATORSaleh Sargolzaei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62466
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03190

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient Boosting within a Single Attention Layer Thanks: Code available at https://github.com/salehsargolzaee/boosted-attention Saleh Sargolzaei Abstract Transformer attention computes a single softmax-weighted average over values—a one-pass estimate that cannot correct its own errors. We introduce gradient-boosted attention , which applies the principle of gradient boosting within a single attention layer: a second attention pass, with its own learned projections, attends to the prediction error of the first and applies a gated correction. Under a squared reconstruction objective, the construction maps onto Friedman’s gradient boosting machine, with each attention pass as a base learner and the per-dimension gate as the shrinkage parameter. We show that a single Hopfield-style update erases all query information orthogonal to the stored-pattern subspace, and that further iteration under local contraction can collapse distinct queries in the same region to the same fixed point. We also show that separate projections for the correction pass can recover residual information inaccessible to the shared-projection approach of Tukey’s twicing. On 10M-token subsets of WikiText-103 and OpenWebText, gradient-boosted attention improves test perplexity by 6.0 % 6.0\% and 5.6 % 5.6\% over standard attention, outperforming both Twicing Attention and a parameter-matched wider baseline on both benchmarks, with two rounds capturing most of the benefit. We further show, both theoretically and empirically, that the mechanism requires the additive residual structure of Pre-LN transformers: under Post-LN, the same architecture degrades perplexity by 9.6 % 9.6\% . 1 Introduction The attention mechanism ( 21 ) computes a softmax-weighted combination of value vectors, conditioned on query-key similarities. This is a single-pass operation: the query is compared against all keys once, a probability distribution is formed, and values are averaged accordingly. When this estimate is poor, whether because the query is ambiguous, the relevant keys are diluted among distractors, or the softmax assigns weight to incompatible values, there is no within-layer mechanism to detect or correct the error. A parallel limitation has long been understood in classical machine learning. A single regression tree or a single kernel smoother produces a biased estimate; the bias can be reduced by fitting a second model to the residual of the first. This is the insight behind gradient boosting ( 8 ) , which builds an additive model F=f_{0}+\eta_{1}f_{1}+\eta_{2}f_{2}+\cdots by sequentially fitting each f_{m} to the negative gradient of the loss with respect to the current prediction. With strong base learners, two or three rounds often suffice ( 8 ) . We apply this principle within a single attention layer. Round 0 produces an initial estimate via standard attention. The residual—the difference between the input and this estimate—is then passed through a second attention round with its own learned \mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V} projections, where all three projections operate on the residual signal. A per-dimension learned gate controls how much of the correction to apply. Crucially, the second round derives its keys and values from the residual, allowing it to operate in a different subspace than the first round and recover information that shared-kernel corrections cannot amplify. The result is a drop-in replacement for standard attention that adds one extra set of projections and a small gating network (approximately 18% additional parameters overall). Why not simply iterate attention? A natural alternative is to iterate the same attention operation: given output \hat{y} , feed it back as the query and repeat. This corresponds to running the modern Hopfield network ( 17 ) toward its fixed point. We show (Section 4 , Proposition 1 ) that this approach systematically destroys query information. Under the local contraction conditions established by 17 , queries in the same contraction region converge to the same fixed point, which is determined by the stored patterns and temperature alone. This makes iteration appropriate for content-addressable memory (where the goal is to retrieve a stored pattern) but harmful for the transformer’s actual task (where the query carries information that must be preserved in the output). Our negative results—including training with Deep Equilibrium Models (DEQ) ( 3 ) —confirm that the iterative approach failed under all training procedures we tested (Section 5 ). Gradient-boosted attention avoids this failure mode by feeding a different signal (the residual) through different projections, rather than re-processing the same state through the same function. Relation to prior work. The idea that transformers implement a form of gradient descent is not new. 5 proved that each transformer layer implements one step of functional gradient descent in a reproducing kernel Hilbert space—which is, mathematically, one round of gradient boosting—though they did not make this connection to the boosting literature explicit. 1 applied Tukey’s twicing ( 20 ) within each attention layer, smoothing the residual \mathbf{V}-\mathbf{A}\mathbf{V} with the same attention matrix \mathbf{A} . Their correction reuses the same attention kernel, yielding (2\mathbf{A}-\mathbf{A}^{2})\mathbf{V} , without learned gating or separate projections for the correction pass. Differential Transformer ( 23 ) computes two attention maps in parallel and subtracts them, canceling shared noise; this is a parallel subtractive mechanism, while ours is sequential and error-corrective. We discuss these and other related works in detail in Section 7 . Contributions. 1. We introduce gradient-boosted attention, a multi-round attention mechanism in which each round corrects the prediction error of previous rounds using separate learned projections and a per-dimension gate. The architecture maps directly onto Friedman’s Multiple Additive Regression Trees (MART) framework. 2. We identify a fundamental limitation of iterative retrieval: query information orthogonal to the stored patterns is erased after one step, and further iteration collapses distinct queries to the same fixed point (Proposition 1 ). We establish a formal correspondence to Friedman’s gradient boosting under a reconstruction objective (Proposition 2 ), and show that shared-projection corrections like Twicing are confined to the row span of the original value matrix and are not fitted to the residual; gradient-boosted attention’s separate projections escape both constraints (Proposition 3 ). 3. We show that Post-LN placement disrupts the additive structure required by gradient boosting: the well-known rank- (d{-}2) LayerNorm Jacobian ( 22 ) couples dimensions and rescales corrections by a data-dependent factor, breaking the element-wise gate control (Proposition 4 ). Empirically, gradient-boosted attention improves perplexity by 6.0\% under Pre-LN but degrades it by 9.6\% under Post-LN, while a parameter-matched wider baseline improves under both placements. 4. On 10M-token subsets of WikiText-103 and OpenWebText, gradient-boosted attention improves test perplexity by 6.0\% and 5.6\% over standard attention, outperforming Twicing Attention and a parameter-matched wider baseline on both benchmarks, with two rounds capturing most of the benefit. 2 Background 2.1 Gradient Boosting Gradient boosting ( 8 ) constructs an additive model F_{M}(x)=\sum_{m=0}^{M}\eta_{m}f_{m}(x) by sequential residual fitting. Starting from an initial estimate F_{0}(x)=f_{0}(x) , each subsequent base learner f_{m} is fit to the negative gradient of the loss: r_{m}=-\frac{\partial L(y,F)}{\partial F}\bigg|_{F=F_{m-1}(x)},\qquad f_{m}\approx r_{m},\qquad F_{m}=F_{m-1}+\eta_{m}f_{m}, (1) where \eta_{m}\in(0,1] is a shrinkage parameter that regularizes the update. For the squared loss L=\frac{1}{2}\|y-F\|^{2} , the negative gradient is simply the residual r_{m}=y-F_{m-1}(x) . A key empirical finding is that with strong base learners (e.g., deep trees), two to three rounds often capture most of the achievable improvement, with rapidly diminishing returns thereafter. 2.2 Attention as Hopfield Retrieval 17 showed that transformer attention is mathematically equivalent to one step of the modern continuous Hopfield network. Given stored patterns \mathbf{X}=[\mathbf{x}_{1},\ldots,\mathbf{x}_{N}]\in\mathbb{R}^{d\times N} and a query \boldsymbol{\xi}\in\mathbb{R}^{d} , the update rule is:

Read the complete text