Delta Attention Residuals
Cheng Luo, Zefan Cai, Junjie Hu · 2026
Read Delta Attention Residuals by Cheng Luo, Zefan Cai, Junjie Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18855.txt.
Introduction
Cheng Luo, Zefan Cai and Junjie Hu rework how transformer layers share information across depth. Standard attention residuals route cumulative hidden states that grow redundant, so attention flattens in deep layers and stops picking useful states. Their Delta Attention Residuals instead route the change each sublayer introduces, which stays varied and sharpens attention markedly. Across models from 220 million to 7.6 billion parameters the approach improves validation perplexity by 1.7 to 8.2 percent and can be fitted to pretrained models by ordinary fine-tuning.
Catalog record
| CREATOR | Cheng Luo, Zefan Cai, Junjie Hu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59508 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18855
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Delta Attention Residuals Cheng Luo, Zefan Cai, Junjie Hu Abstract Attention Residuals replace standard additive residual connections with learned softmax attention over previous layer outputs, enabling selective cross-layer routing. However, standard Attention Residuals still attend over cumulative hidden states in previous layers, which are highly redundant. We show that this redundancy leads to routing collapse in deeper layers: attention weights become low-contrast and closer to uniform (max weight ≈ {\approx} 0.2), limiting the model’s ability to select informative states in previous layers. This raises a key but underexplored design question: what layer-wise representations should be routed in Attention Residuals ? To answer this question, we propose Delta Attention Residuals , which attend over deltas —the change introduced by each sublayer ( 𝐯 i = 𝐡 i + 1 − 𝐡 i \mathbf{v}_{i}=\mathbf{h}_{i+1}-\mathbf{h}_{i} )—instead of cumulative states. Delta representations are structurally diverse and yield higher-contrast attention distributions (max weight ≈ {\approx} 0.6), enabling more selective and effective routing across layers. This principle applies at both per-sublayer and block granularity. Across all tested scales (220M–7.6B), Delta Attention Residuals consistently outperform both standard residuals and Attention Residuals, with 1.7–8.2% validation perplexity gains. Delta Attention Residuals also enables converting pretrained checkpoints into Delta Attention Residuals via standard fine-tuning. Code is available at https://github.com/wdlctc/delta-attention-residuals-code . 1 Introduction Residual connections ( 10 ) are fundamental to training deep transformers ( 29 ) . The standard update, \mathbf{h}_{l}=\mathbf{h}_{l-1}+f_{l}(\mathbf{h}_{l-1}) , accumulates all preceding sublayer outputs with fixed additive coefficients. While this update provides gradient highways ( 30 ) , it has no mechanism to selectively aggregate the preceding layers across depth. Attention Residuals ( 15 ) address this issue by replacing fixed aggregation with learned softmax attention over prior layer outputs ( \mathbf{h}_{0},\mathbf{h}_{1},\ldots,\mathbf{h}_{1-1} ), allowing selective residuals routed from prior layers to the current layer. However, as each state \mathbf{h}_{l} in Attention Residuals is still a running sum of all previous layer outputs \mathbf{h}_{0:l-1} , adjacent states \mathbf{h}_{l},\mathbf{h}_{l-1} become highly redundant. As depth increases, the pairwise similarity between adjacent states grows, reducing the contrast among routing candidates. Under such low-contrast sources, softmax attention becomes less discriminative, leading to routing collapse , where attention weights approach a near-uniform distribution. Empirically, we observe that the maximum routing weight drops to about 0.2 in deep layers (Figure 1 a), indicating that the mechanism loses its ability to meaningfully select among sources and instead averages over them. This raises a critical yet underexplored design question: what layer-wise sources should be routed to the current layer? We observe that the change each layer introduces is far more informative than the cumulative states. The delta \mathbf{v}_{i}=\mathbf{h}_{i+1}-\mathbf{h}_{i} captures what a specific sublayer contributed , not where the model has been . Adjacent deltas are naturally diverse because delta outputs serve different functions and operate in different subspaces ( 7 ) , while cumulative states converge to near-linear relationships across layers ( 24 ) . We propose Delta Attention Residuals , which route over these deltas instead of cumulative states. The same principle applies at two granularities: per-sublayer deltas (each attention or MLP output individually) and block-level deltas (the accumulated change over a group of layers). Delta routing uses an additive formulation (\mathbf{h}=\tilde{\mathbf{h}}+\sum\alpha_{i}\mathbf{v}_{i}) . This enables sharp, selective cross-layer shortcuts with max softmax weight increased to {\sim} 0.6 (Figure 1 a) and the additive formulation preserves the residual stream and enables finetuning of pretrained models with zero initialization disruption. Our contributions are: Figure 1 : Source redundancy in cross-layer routing (Qwen3-0.6B, L{=}28 ). (a) Per-layer routing sharpness: AttnRes routing degrades to max weight {\sim} 0.2 in deep layers, while Delta Block maintains sharp routing ( {\sim} 0.6). (b) Routing quality: Delta Block achieves 1.8{\times} higher average max weight (0.62 vs. 0.35). (c) Training loss: Delta Block (green) consistently outperforms AttnRes (red). 1. We identify the routing collapse problem in Attention Residuals: source redundancy causes routing sharpness to degrade to max weight {\sim} 0.2 in deep layers, rendering the mechanism near-uniform (§ 2.2 ). 2. We propose Delta Attention Residuals , which route over deltas ( \mathbf{v}_{i}=\mathbf{h}_{i+1}-\mathbf{h}_{i} ) instead of cumulative states, maintaining sharp routing (max weight {\sim} 0.6) and consistently improving over both baseline and AttnRes from 220M to 7.6B parameters Qwen-based model, (§ 3 ). 3. We show that delta routing’s additive formulation enables the easy conversion of existing pretrained transformers into Delta Attention Residuals via fine-tuning. Fine-tuned Delta Block outperforms their pretrained transformer checkpoints on 8 downstream benchmarks (§ 3.5 ). 2 Method: Delta Attention Residuals 2.1 Preliminaries: Attention Residuals A Pre-Norm transformer ( 29 ; 33 ) updates the hidden state as \mathbf{h}_{i+1}=\mathbf{h}_{i}+\mathbf{v}_{i} , where \mathbf{v}_{i}:=f_{i}(\mathbf{h}_{i}) is defined as the output of sublayer i —here a sublayer can be either an attention or MLP layer. Equivalently, \mathbf{v}_{i}=\mathbf{h}_{i+1}-\mathbf{h}_{i} also measures the change introduced at depth i . Standard residuals accumulate all sublayer outputs with fixed unit coefficients. Attention Residuals ( 15 ) replace this with a learned weighted sum: \hat{\mathbf{h}}_{l}=\sum_{i=0}^{l-1}\alpha_{i\to l}\cdot\mathbf{s}_{i},\quad\alpha_{i\to l}=\frac{\exp(\mathbf{w}_{l}^{\top}\mathrm{RMSNorm}(\mathbf{s}_{i}))}{\sum_{j}\exp(\mathbf{w}_{l}^{\top}\mathrm{RMSNorm}(\mathbf{s}_{j}))} (1) where \mathbf{s}_{i} are the source representations from preceding layers i\in[0,l) , \mathbf{w}_{l}\in\mathbb{R}^{d} is a learned query (zero-initialized), and \hat{\mathbf{h}}_{l} feeds into the next sublayer. A critical question arises: what should \mathbf{s}_{i} be? 2.2 The Source Redundancy Problem Figure 2 : Architecture comparison. (a) Standard Residuals: uniform additive accumulation with fixed unit coefficients. (b) Attention Residuals ( 15 ) : learned softmax attention (weights \mathbf{w} , aggregation a ) over cumulative hidden states, but source redundancy degrades routing to max weight {\sim} 0.2 in deep layers. (c) Delta Attention Residuals (ours): attends over per-sublayer delta outputs, maintaining sharp routing (max weight {\sim} 0.6) via additive routing. Compatible with MLP layers. We consider two source representations for \mathbf{s}_{i} in Eq. 1 : Cumulative sources. \mathbf{s}_{i}=\mathbf{h}_{i}=\mathbf{h}_{0}+\sum_{j=1}^{i}\mathbf{v}_{j} —the full hidden state after sublayer i . Since each \mathbf{h}_{i} is a running sum, adjacent states share an increasingly large common prefix as depth grows, so the softmax logits \mathbf{w}^{\top}\mathrm{RMSNorm}(\mathbf{h}_{i})\approx\mathbf{w}^{\top}\mathrm{RMSNorm}(\mathbf{h}_{i+1}) and the routing distribution approaches uniform. Empirically, at Qwen3-0.6B scale ( L{=}28 ), the max softmax weight drops to {\sim} 0.2 in deep layers (Figure 1 a), rendering the mechanism near-uniform. Delta sources. \mathbf{s}_{i}=\mathbf{v}_{i} —the per-sublayer output defined in § 2 . At the finest granularity, each attention output and each MLP output is a separate source ( Delta AttnRes , 2L sources for L layers). Adjacent deltas are structurally diverse because attention and MLP outputs occupy different subspaces, and outputs from different depths capture different abstraction levels. This naturally coarsens to block-level grouping: a block delta \Delta_{b}=\mathbf{h}_{b+1}-\mathbf{h}_{b} aggregates multiple sublayer outputs into one source ( Delta Block , B{+}1 sources for B blocks), trading granularity for efficiency. See Appendix C for a detailed schematic of the block variant. 2.3 Delta Attention Residuals Delta Attention Residuals use additive routing : rather than replacing the residual stream with a weighted combination of cumulative states, we add selected delta information: \hat{\mathbf{h}}_{l}=\tilde{\mathbf{h}}_{l}+\sum_{i}\alpha_{i\to l}\cdot\mathbf{v}_{i} (2) where \mathbf{v}_{i} are per-sublayer outputs, \tilde{\mathbf{h}}_{l} is the current residual stream, and \alpha_{i\to l}=\mathrm{softmax}(\mathbf{w}_{l}^{\top}\mathrm{RMSNorm}(\mathbf{v}_{i})) . This formulation has three advantages: 1. Residual preservation. The residual stream \tilde{\mathbf{h}}_{l} is preserved by default; routing adds information rather than replacing it. 2. No information loss at block boundaries. In cumulative-state AttnRes, only the states at block boundaries are retained as routing sources—intermediate sublayer contributions within each block are collapsed