Learning to Focus: Focal Attention for Selective and Scalable Transformers
Dhananjay Ram, Wei Xia, Stefano Soatto · 2025
Read Learning to Focus: Focal Attention for Selective and Scalable Transformers by Dhananjay Ram, Wei Xia, Stefano Soatto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06818.txt.
Introduction
Dhananjay Ram, Wei Xia, and Stefano Soatto propose a small change to the attention mechanism at the heart of transformer models. Standard softmax attention often spreads its focus too thinly, producing noisy distributions that blur feature selection, especially over long contexts. Their Focal Attention sharpens that distribution by adjusting the softmax temperature.
The temperature can be fixed in advance or learned during training, letting the model concentrate on relevant tokens and suppress the rest. Across benchmarks, models with Focal Attention matched standard transformers using up to 42 percent fewer parameters or a third less training data. On long-context tasks it delivered relative gains between 17 and 82 percent.
Catalog record
| CREATOR | Dhananjay Ram, Wei Xia, Stefano Soatto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06818
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Focus: Focal Attention for Selective and Scalable Transformers Dhananjay Ram, Wei Xia, Stefano Soatto Abstract Attention is a core component of transformer architecture, whether encoder-only, decoder-only, or encoder–decoder model. However, the standard softmax attention often produces noisy probability distribution, which can impair effective feature selection at every layer of these models, particularly for long contexts. We propose Focal Attention , a simple yet effective modification that sharpens the attention distribution by controlling the softmax temperature, either as a fixed hyperparameter or as a learnable parameter during training. This sharpening enables the model to concentrate on the most relevant tokens while suppressing irrelevant ones. Empirically, Focal Attention scales more favorably than standard transformer with respect to model size, training data, and context length. Across diverse benchmarks, it achieves the same accuracy with up to 42% fewer parameters or 33% less training data. On long-context tasks, it delivers substantial relative improvements ranging from 17% to 82%, demonstrating its effectiveness in real world applications. 1 Introduction The recent progress in artificial intelligence (AI) has been largely driven by the development of high-quality large language models (LLMs) ( 1 ; 17 ; 33 ) . These LLMs are trained with transformer architecture ( 54 ; 5 ) and attention mechanism lies at the center of transformer model, which enables dynamic weighting of input tokens based on contextual relevance. Attention mechanism computes the importance of each token in the context of a sequence of tokens and is implemented using softmax function. The output of the softmax can be noisy, resulting in assigning some probability mass to even irrelevant tokens as shown in Figure 1 . Figure 1: Comparison of attention probability distribution of a baseline transformer model and the proposed Focal Attention . We see that the Focal Attention reduces the attention noise, and redistributes the attention probability from irrelevant tokens to relevant ones. A common strategy to denoise the softmax outputs is to introduce a temperature parameter that scales the logits before applying the softmax transformation. Lower temperatures yield more peaked distributions, while higher values result in flatter distributions. This technique has been used at the output layer of LLMs to generate diverse samples during inference ( 22 ; 60 ) , and has shown promise in improving performance through selective sampling and test-time inference strategies ( 50 ; 40 ) . Despite its utility, temperature scaling has not been explored for attention mechanism in the internal layers of Transformer models. In the original Transformer ( 54 ) the attention logits are implicitly scaled by the inverse square root of the input dimensionality ( 54 ) , which acts as a constant temperature. However, this uniform scaling may not be optimal across all contexts and layers, specially for long-context tasks where identifying relevant information from among thousands (or even hundreds of thousands) of tokens is inherently challenging ( 34 ) . In this work, we explore the role of temperature in attention mechanism and propose Focal Attention , where the temperature can be set as a hyperparameter or learned during model training. Setting a relatively smaller value for the temperature, the model sharpens the attention distribution and enables more focused selection of relevant tokens. This can improve feature selection across layers, and enhance the model’s ability to reason over long contexts. Our contributions can be summarized as follows: Focal Attention We propose to explicitly control the softmax temperature within the attention mechanism to generate sharper attention distributions and enable more selective feature extraction (Section 2 ). Improved Performance We demonstrate that the proposed Focal Attention consistently outperforms the standard Transformer model across a range of downstream tasks (Section 4.2 ). Scaling Properties We show that our approach scales more effectively with model size, training tokens, and context length compared to standard Transformers (Section 4.1 ). Long-Context Superiority Focal Attention offers significant gains in long-context scenarios, particularly in settings such as in-context learning, retrieval-augmented generation, and information retrieval, making it highly applicable to real-world applications (Section 4.3 ). 2 Focal Attention We propose Focal Attention as an alternative to the attention layer used in transformer models, e.g. GPT ( 5 ) , LLaMA ( 53 ) , Qwen ( 2 ) , Deepseek ( 33 ) , etc family of models. This attention is particularly useful for longer context modeling due to its ability to reduce noise in the attention probability and enables better feature selection at every layer. The standard softmax function is defined as: P_{i}=\frac{exp(z_{i})}{\sum_{j=1}^{n}(exp(z_{j}))} (1) When introducing the temperature parameter t , the logits are divided by t before applying the softmax: P_{i}=\frac{exp(z_{i}/t)}{\sum_{j=1}^{n}(exp(z_{j}/t))} (2) Here, z_{i} is the i-th logit (input to softmax) and t>0 is the temperature. The value t>1 reduces the magnitude of differences between the logits, making the softmax output more uniform (flatter distribution) and less certain in any single class. This can be useful for encouraging exploration or increasing diversity in sampling (e.g., in text generation). On the other hand, t<1 increases the magnitude of differences between the logits, resulting in the softmax output more peaked (concentrated distribution). The highest logit dominates the probability mass, leading to more confident predictions or less diverse sampling. 2.1 Transformer with Constant Temperature Let X=[X_{1},X_{2},\ldots,X_{n}] be the input sequence to the transformer model. Here we consider decoder-only architecture as an example to show Focal Attention . The model consists of a multi-layer architecture where each layer has an attention module followed by a Feed-forward module ( 54 ; 53 ) . Our proposed Focal Attention updates the attention module, keeping other parts of the model same. Any given input X to the attention module is projected into the query ( Q ), key ( K ) and value ( V ) space. The Query and Key representations are used to compute the attention probability, which is multiplied with the Value representation to yield the output. Q=XW^{Q},K=XW^{K},V=XW^{V} (3) Attention(X)=softmax{(\frac{QK^{T}}{\sqrt{d}})}V (4) where W^{Q} , W^{K} , W^{V} are parameter matrices. The softmax temperature is \sqrt{d} and it is shown to produce noisy probability distribution ( ) , as shown in Figure 1 . In this work, we propose to reduce the temperature by using a scaling factor t<1 as a hyperparameter to the model: Attention(X)=softmax{(\frac{QK^{T}}{t\sqrt{d}})}V (5) This produces a sharper probability distribution enabling the model to better focus on relevant tokens and ignore the irrelevant ones. We can use a different scaling factor per layer, however we use a constant scaling for the whole model. 2.2 Transformer with Learned Temperature We propose to further generalize the previous approach by learning the temperature ( \tau ) from the hidden representation. This enables the model to learn different scaling factors for different layers in the model, which is desirable because lower layers may require softer attention probability distribution and higher layers may require sharper attention probability distribution as the model get more confident. More specifically, we learn the softmax temperature using the input to the attention module ( X\in\mathbb{R}^{n\times d} ) of the model by learning a parameters vector ( w_{\tau}\in\mathbb{R}^{d} ) of the size of hidden states in every layer as shown in Eq 6 \tau=clip(mean(Xw_{\tau}),\tau_{min},\tau_{max}) (6) We clip the temperature value to a predefined minimum ( \tau_{min} ) and maximum ( \tau_{max}