TRA: Better Length Generalisation with Threshold Relative Attention
Mattia Opper [email protected], Roland Fernandez [email protected], Paul Smolensky [email protected], Jianfeng Gao [email protected] · 2025
Read TRA: Better Length Generalisation with Threshold Relative Attention by Mattia Opper [email protected], Roland Fernandez [email protected], Paul Smolensky [email protected], Jianfeng Gao [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23174.txt.
Introduction
Mattia Opper and co-authors probe why transformers fail when text runs longer than anything they trained on. Even simple tasks collapse at new lengths. Their 2025 paper blames two faults in the attention mechanism, the part that decides which words matter to which. First, attention never fully erases irrelevant words. Second, learned position biases can accidentally boost a distant, irrelevant word just because of where it sits.
Their fix, Threshold Relative Attention, attacks both faults. A sparsity rule cuts irrelevant keys out of the attention math entirely, rather than merely down-weighting them. Distance is then judged only among the keys that survive, so far-off clutter cannot sneak back in. Rebuilt this way, decoder-only transformers generalized to new lengths far better in the team's tests. The work suggests the length problem is a fixable flaw in how attention handles position and noise, not a deep mystery of scale.
Catalog record
| CREATOR | Mattia Opper [email protected], Roland Fernandez [email protected], Paul Smolensky [email protected], Jianfeng Gao [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23174
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TRA: Better Length Generalisation with Threshold Relative Attention Thanks: Work completed while at Microsoft Research Mattia Opper [email protected], Roland Fernandez [email protected], Paul Smolensky [email protected], Jianfeng Gao [email protected] Abstract Transformers struggle with length generalisation, displaying poor performance even on basic tasks. We test whether these limitations can be explained through two key failures of the self-attention mechanism. The first is the inability to fully remove irrelevant information. The second is tied to position, even if the dot product between a key and query is highly negative (i.e. an irrelevant key) learned positional biases may unintentionally up-weight such information - dangerous when distances become out of distribution. Put together, these two failure cases lead to compounding generalisation difficulties. We test whether they can be mitigated through the combination of a) selective sparsity - completely removing irrelevant keys from the attention softmax and b) contextualised relative distance - distance is only considered as between the query and the keys that matter . We show how refactoring the attention mechanism with these two mitigations in place can substantially improve the generalisation capabilities of decoder only transformers. 1 1 1 Pytorch code for core mechanism in Appendix A . Reviewed on OpenReview: https://openreview.net/forum?id=yNiBUc2hMW 1 Introduction Transformers ( Vaswani et al., 2017 ) have sparked a revolution in artificial intelligence. When pre-trained at scale, these models display impressive broad-spectrum capabilities, particularly in the domains of natural language and code ( Bubeck et al., 2023 ; Dubey et al., 2024 ) . This includes the ability to perform complex abstract symbolic processing through in-context learning (ICL) ( Brown et al., 2020 ; Sinha et al., 2024 ; Smolensky et al., 2024 ) . On the other hand, these abilities are far from robust ( Dziri et al., 2023 ; Mirzadeh et al., 2024 ) and display dramatic failure modes such as self-contradiction and hallucination ( Liu et al., 2023a ; Ji et al., 2023 ; Guerreiro et al., 2023 ) . Similar failure modes are also observed in controlled synthetic settings where the correct solution should be trivial . Decoder-only transformers consistently display poor generalisation to unseen lengths and dependency distances ( Zhou et al., 2023 ; Liu et al., 2023a ) . Addressing these issues can provide crucial benefits regarding both long context utilisation and length generalisation more broadly. Firstly, current approaches generally involve training at a certain sequence length and then adapting the positional encoding to longer context. Most commonly this entails additional finetuning for best effect, thereby incurring a further compute cost penalty. Even then current SoTA foundation models still struggle with long context ( Hsieh et al., 2024 ) . Ideally, we would want a solution that simply works out the box. Put another way, if the transformer is in fact learning circuits ( Merrill et al., 2022 ; Olsson et al., 2022 ; Liu et al., 2023b ) , then their correct execution should be general purpose and robust, rather than contingent on specific quirks of the training data. To understand how to get there, we should first look at the core capability that the attention mechanism was supposed to enable in the first place, namely the ability to selectively retrieve and propagate information ( Bahdanau et al., 2014 ; Luong et al., 2015 ; Vaswani et al., 2017 ) . We can break this capability down into three broad categories. (1) Purely positional: attending to ones local neighbourhood (i.e. contextualisation within a sentence or phrase). (2) Purely based on content: attending to the most relevant item irrespective of position. (3) Content followed by position: of the relevant items apply a positional bias (e.g. select the most recent). (3) is a crucial capability, it enables tracking the state of an entity over time, and is a pre-requisite for robust reasoning. Our contention is that standard attention mechanisms can perform (1) and (2), but struggle with (3). This due to the way that semantic content and positional information combine. Position is generally treated as independent of content. Which means that favouring one must come at the expense of the other. This leaves the two types of information in conflict, fundamentally limiting self-attention. Our solution involves a two step process. We first mask out irrelevant keys based on threshold applied to the raw attention weights, and then compute the relative distance but only as between the query and the keys that remain. We call this mechanism T hreshold R elative A ttention ( TRA ). TRA is a simple, modular addition that allows self-attention to learn general solutions to all three behaviours outlined above. We find that TRA’s introduction leads to considerably improved length generalisation on synthetic benchmarks. As well as robust, and even improved, language modeling perplexity for out-of-distribution sequence lengths, even up to an increase of 32x. Our findings contribute to the development of more robust attention mechanisms. 2 Model 2.1 Background: Relative Positional Encodings Relative positional encodings ( Dai et al., 2019 ; Raffel et al., 2019 ) decompose the attention operation into two steps. First the standard {\bm{Q}}{\bm{K}}^{T} dot product is calculated, but this purely based on semantic relevance. Let’s call that matrix {\bm{S}} (semantic). This is then added with a learned positional bias matrix {\bm{D}} (distance) to produce the final attention weight. \displaystyle{\bm{S}} \displaystyle=\frac{{\bm{Q}}{\bm{K}}^{T}}{\sqrt{d_{\bm{k}}}} (1) \displaystyle{\bm{A}} \displaystyle=\text{softmax}\left({\bm{S}}+{\bm{D}}\right) (2) Our contention is that a central cause of length generalisation issues stems from {\bm{S}} and {\bm{D}} being treated as independent of each other. If {\bm{D}} is fit to the training distribution, then distribution shifts will necessarily begin to introduce errors. {\bm{S}} is more robust in this regard, because relevance isn’t impacted by changing lengths. We want to make {\bm{D}} more robust by making it contingent on {\bm{S}} . 2.2 Contextual Relative Distance We want to condition {\bm{D}} on {\bm{S}} . In other words we only want to consider position over keys that matter. To achieve this we first introduce selective sparsity . We will allow certain keys to be fully removed from {\bm{S}} , but do not force it to be one hot. Instead we introduce a threshold and eliminate all keys which fall below it. Lets call that matrix \bm{S^{\prime}} . The simplest way to get \bm{S^{\prime}} is by applying a ReLU. \displaystyle\bm{S^{\prime}} \displaystyle=\text{ReLU}\left(\frac{{\bm{Q}}{\bm{K}}^{T}}{\sqrt{d_{\bm{k}}}}\right) (3) Next calculate a boolean mask {\bm{M}} that is true only for relevant positions: \displaystyle{M}_{i,j} \displaystyle=\begin{cases}1&\text{if }S^{\prime}_{i,j}>0\\ 0&\text{otherwise}\end{cases} (4) We are now going to compute our contextualised distance matrix, but only consider the relative distance to keys which make it through the threshold, this give us the contextualised relative distance . To do this we consider the following example {\bm{M}} , where again only keys which meet the threshold are assigned 1: \displaystyle{\bm{M}} \displaystyle=\begin{bmatrix}1&0&0&0\\ 1&0&0&0\\ 0&1&1&0\\ 1&0&1&0\\ \end{bmatrix} (5) Our threshold mask {\bm{M}} is a m\times n matrix, which we are going to use to get our contextualised relative distance matrix \bm{\bar{D}} , by applying the cumsum operation in the right to left direction (i.e. producing relative positions given causal masking). More formally we can say that for row index i and column index j: \displaystyle\bar{D}_{ij}={\sum}_{k=j}^{n}M_{ik} (6) For our example {\bm{M}} from (5) this would yield the following output, noting that we omit the keys which do not meet the threshold in this example for clarity: