Exclusive Self Attention
Shuangfei Zhai · 2026
Read Exclusive Self Attention by Shuangfei Zhai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09078.txt.
Introduction
Shuangfei Zhai offers exclusive self attention, a simple tweak of self attention for Transformer models. The key idea limits attention to signals at right angles to the token's own value vector, dropping self-position signals to sharpen context modeling. Tested on standard language modeling, exclusive attention always beats plain attention across models up to 2.7B, with gains growing as sequences grow.
Catalog record
| CREATOR | Shuangfei Zhai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13511 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09078
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exclusive Self Attention Shuangfei Zhai Abstract We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer’s sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token’s own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows. 1 Introduction Transformers ( 13 ) consist of interleaved self attention (SA) and feed forward (FFN) layers, where SA aggregates information from the context, and FFN performs position wise feature updates. This SA/FFN design has remarkably stood the test of time and is continuing to serve as the building block for modern Transformer variants. In this work, we hypothesize that it is beneficial to further promote the division of labor between SA and FFN. We first expose a peculiar behavior of Transformers, where the output of attention tends to have a high cosine similarity with the self value vector – and we call it the attention similarity bias . The prevalence of the attention similarity bias suggests that SA spends a significant portion of its capacity modeling the point wise feature transformation. This is on the one hand unnecessary, because the information of the current position has a direction residual path to the following FFN layer; and on the other hand harmful, because it creates a competition between modeling the contextual vs point-wise feature. This reasoning directly motivates our solution: exclusive self attention (XSA), which explicitly excludes directions from the attention’s output along that of the self value vector. We evaluate XSA against standard Transformers on the language modeling task, and we show that XSA 1) introduces minimal computational overhead; 2) achieves better training/validation loss across three model sizes; 3) achieves better downstream evaluation results; 4) maintains consistent gains across different learning rates; 5) shows larger gains as sequence length increases; and 6) is robust w.r.t. the use of attention sinks. Figure 1: Visualization of the attention similarity bias of a 1.3B parameter language model of sequence length 2048 trained for 100B tokens, aggregated on 1024 random training sequences. Left : the average cosine similarity of value vectors v_{i} , v_{j} within a sequence; middle : the average diagonal attention value a_{i,j} ; right : the average cosine similarity of attention output y_{i} and the self value vector v_{i} . See Eq. 1 for notations. 2 Motivation We first define a standard (causal) self attention (SA) as y=f(x) : \begin{split}&q_{i}=W_{q}x_{i},\;k_{j}=W_{k}x_{j},\;v_{j}=W_{v}x_{j},\;a_{i,j}=\frac{\exp(q_{i}^{T}k_{j})}{\sum_{j^{\prime}=1}^{i}\exp(q_{i}^{T}k_{j^{\prime}})},\;y_{i}=\sum_{j=1}^{i}a_{i,j}v_{j},\end{split} (1) where W_{q},W_{k},W_{v} are the query, key and value projections, respectively. We next demonstrate the attention similarity bias phenomenon, which exposes a hidden problem of SA. To see this, we take a trained language model (which has 1.3B parameters and 2048 sequence length, see Sect. 4 for details) and analyze each of its attention layers in Figure 1 . Specifically, we plot three quantities: 1) the average cosine similarity of value vectors <v_{i},v_{j}> within the same head and sequence; 2) the average diagonal values of the attention a_{i,i} ; and 3) the average cosine similarity of the aggregated values y_{i} with the corresponding value v_{i} . All quantities are then averaged across attention heads and 1024 random training sequences and plotted for each layer. We can see that 1) value vectors tend to be positively correlated and 2) attention scores to the current position are relatively high. As a direct consequence, there is a high average <y_{i},v_{i}> , with an increasing trend w.r.t. the layer index. This suggests that standard SA tends to aggregate value vectors similar to what the self value v_{i} already encodes, which implicitly overlaps with role of FFN and consequently diminishes SA’s goal of context modeling. 3 Method We define exclusive self attention (XSA) as z=f(x) : \begin{split}&q_{i}=W_{q}x_{i},\;k_{j}=W_{k}x_{j},\;v_{j}=W_{v}x_{j},\;a_{i,j}=\frac{\exp(q_{i}^{T}k_{j})}{\sum_{j^{\prime}=1}^{i}\exp(q_{i}^{T}k_{j^{\prime}})},\;y_{i}=\sum_{j=1}^{i}a_{i,j}v_{j},\\ &{\color[rgb]{0,0,1}z_{i}=y_{i}-(y_{i}^{T}v_{i})\frac{v_{i}}{\|v_{i}\|^{2}_{2}}}.\;\end{split} (2) Note that the first line of Equation 2 corresponds to standard SA in Equation 1 ; XSA introduces an additional step to remove the projection of SA’s output y_{i} on self value vector v_{i} . Consequently, XSA’s output z_{i} no longer contains v_{i} itself, nor components from that context that’s correlated with v_{i} . Therefore, XSA completely removes the attention similarity bias . Our core hypothesis is that, in the presence of residual connections and the FFN block, XSA 1) maintains the expressiveness of standard SA and 2) promotes modeling efficiency by allowing the attention layer to exclusively focus on contextual information. While it is also possible to provide theoretical groundings for it, in this work we defer to empirical evaluations as the main justification. XSA can be implemented with two lines of code change on top of SA, as demonstrated in Algorithm 1 . Algorithm 1 PyTorch-style pseudocode for multi-head causal XSA ⬇ # x: (B,T,D) # Wq, Wk, Wv, Wo: (D,D) # H: number of heads def exclusive_self_attention ( x , Wq , Wk , Wv , Wo , H ): B , T , D = x . shape # linear projections Q = ( x @ Wq ). reshape ( B , T , H , D // H ). transpose (1, 2) K = ( x @ Wk ). reshape ( B , T , H , D // H ). transpose (1, 2) V = ( x @ Wv ). reshape ( B , T , H , D // H ). transpose (1, 2) # standard multi-head attention Y = torch . nn . functional . scaled_dot_product_attention ( Q , K , V , is_causal = True ) # XSA mode Vn = torch.nn.functional.normalize(V, dim=-1) Z = Y - (Y * Vn).sum(dim=-1, keepdim=True) * Vn # output projection out = Z . transpose (1, 2). reshape ( B , T , D ) @ Wo return out 4 Experiments 4.1 Setup Codebase We conduct all experiments with the NanoGPT 1 1 1 https://github.com/karpathy/nanoGPT codebase due to its ease of reproducibility. A few changes are made to the Transformer implementation: we replaced the learned position embeddings with RoPE ( 12 ) whish is a common practice in modern language models; we insert an additional LayerNorm ( 1 ) right after the token embeddings which is found to improve training stability; we allow the number of attention heads and the head dimension to be configured independently to allow for more flexible model sizes. Dataset We use the FineWeb-100BT ( 8 ) 2 2 2 https://huggingface.co/datasets/HuggingFaceFW/fineweb dataset which contains \sim 100 billion tokens. We preprocess the dataset following the NanoGPT protocol: namely tokenizing it with the GPT2 ( 9 ) tokenizer and randomly splitting 0.05\% of the tokens as the validation set. Training details We closely follow the default training settings of NanoGPT, with a few modifications detailed below. We adopt a default context length of 2048, a global batch size of 256, and a training duration of 200K iterations. This amounts to 100B training tokens which is roughly one epoch over the training set. AdamW ( 5 ) is used with a linear learning rate warmup of 2K steps, followed by a cosine decay schedule to \frac{1}{10} x of the max learning rate. We perform a grid search of learning rate