Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji · 2025
Read Softpick: No Attention Sink, No Massive Activations with Rectified Softmax by Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20966.txt.
Introduction
Zayd Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji introduce Softpick, a small change to how transformer networks pay attention, in a 2025 paper. The standard softmax step forces attention weights to sum to one. That breeds two known glitches. One is attention sink, where tokens dump attention on a meaningless first token. The other is massive activations, extreme values that wreck low-precision arithmetic.
Softpick rectifies the function and drops the sum-to-one rule. In models of 340 million and 1.8 billion parameters, the sink rate fell to zero, attention maps grew sparser, and hidden states became far better behaved. Quantized Softpick models beat their softmax counterparts on standard benchmarks, with the edge largest at the lowest bit widths. The authors argue the change could help quantization, low-precision training, pruning, and interpretability, and they release their code.
Catalog record
| CREATOR | Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20966
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji Abstract We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our experiments with 340M and 1.8B parameter models demonstrate that softpick achieves 0% sink rate consistently. The softpick transformers produce hidden states with significantly lower kurtosis and creates sparse attention maps. Quantized models using softpick outperform softmax on standard benchmarks, with a particularly pronounced advantage at lower bit precisions. Our analysis and discussion shows how softpick has the potential to open new possibilities for quantization, low-precision training, sparsity optimization, pruning, and interpretability. Our code: https://github.com/zaydzuhri/softpick-attention . 1 Introduction The softmax function is widely used in statistics and particularly in machine learning as a way to normalize a vector of real numbers into a probability distribution. It has since been adopted as the de facto function to normalize the scores in the attention mechanism ( Bahdanau et al., 2015 ; Sukhbaatar et al., 2015 ) used in the transformer architecture ( Vaswani et al., 2017 ) . The use of softmax in attention was a natural choice, since it represents the probability of a query "matching" to a key among many keys, which can be used to return values each weighted by that probability. However, our modern use of attention makes us question this intuition ( Miller, 2023 ; Smith, 2025 ) : why must they be non-zero probabilities that sum to one? Figure 1: (Top) Comparison between the attention maps when using softmax vs softpick and overall sink rate of the 340M models. (Bottom) Largest hidden state activation per layer of the 340M models. Softmax has great training stability with a neat Jacobian matrix resulting in a dense gradient, bounded Frobenius norm, and built-in regularization ( Saratchandran et al., 2024 ) , and has been proven to be sufficiently expressive in its non-linearity ( Chiang, 2025 ) . Despite the benefits, softmax brings several oddities detrimental to language modeling. Its sum-to-one nature forces out a now well-observed behavior named attention sink ( Xiao et al., 2024 ) where attention heads allocate a significant score towards a specific token, often the initial BOS token, which itself is semantically irrelevant ( Gu et al., 2024 ) . These sinks are, as far as we know, harmless to downstream performance. However, another symptom of this same effect is the appearance of extreme hidden state activations, dubbed massive activations ( Sun et al., 2024 ) , which grow larger as the model scales ( Bondarenko et al., 2023 ) . These activations are especially problematic for quantization, with entire algorithms built around them. Additionally, most low-precision training approaches try to work around these massive activations as they are unwieldy in low-bit settings. Removing these massive activations would open up more possibilities. Even with these quirks, softmax remains widely used in attention due to its stability and effectiveness. In this paper, we propose the softpick function in an attempt to find a normalization function more fitting for attention and its use in transformers. We mitigate both attention sink and massive activations by reshaping the softmax function to avoid strict sum-to-one behavior and allowing rectified outputs. Our contributions are as follows: 1. We propose the softpick function as a drop-in replacement to softmax in attention. 2. We experiment by training 340M and 1.8B parameter transformer models from scratch and show how softpick compares to softmax in benchmarks and training behavior. We also show that quantized softpick models down to 2-bit precision outperform softmax. 3. We analyze the resulting models and show that softpick returns more legible attention maps, a sink rate of 0%, and no massive activations in the hidden states. 4. We investigate the subpar performance of softpick at the 1.8B scale and provide two possible hypotheses that extend our understanding of softmax-like functions for attention. Figure 2: Training loss and gradient norm during training of 340M models. 2 Background Attention sink was characterized by Xiao et al. (2024) as heads that allocate disproportionate attention to semantically weak but frequent tokens, most commonly the BOS token. Barbero et al. (2025) argue that such sink-like heads can mitigate over-mixing by acting as approximate no-ops, while Gu et al. (2024) systematically study correlates of the phenomenon and report that sinks appear broadly in sufficiently trained transformer language models. Together, these results suggest that sink behavior is not an artifact of a specific setup, but is closely tied to attention normalization, in particular the sum-to-one constraint imposed by softmax. Massive activations, identified by Sun et al. (2024) as rare but extremely large hidden-state values, pose a challenge for quantization and low-precision training ( Dettmers et al., 2022 ) . Sun et al. (2024) further link these outliers to self-attention and softmax-driven sink behavior, motivating mitigation strategies that relax strict normalization. For example, Gu et al. (2024) show that removing normalization reduces sinks, but sinks reappear once normalization is reintroduced. KV bias variants similarly require additional learned parameters and can degrade with depth ( Gu et al., 2024 ; Sun et al., 2024 ) . Softmax+1 relaxations add a constant to the denominator ( Miller, 2023 ) and are evaluated in Kaul et al. (2024) , while Agarwal et al. (2025) adopt a learnable-bias variant at scale to relax strict sum-to-one behavior. However, Owen et al. (2025) find KV bias techniques to be unreliable for mitigating massive activations, and architectural alternatives such as gated attention require additional parameters ( Qiu et al., 2025 ) . We propose softpick as a drop-in alternative that mitigates both sinks and massive activations without adding parameters or custom optimizers. 3 Method 3.1 Softpick Function Given a vector \mathbf{x}\in\mathbb{R}^{N} , we define the softpick function as \operatorname{Softpick}(\mathbf{x})_{i}=\frac{\operatorname{ReLU}(e^{x_{i}}-1)}{\sum_{j=1}^{N}|e^{x_{j}}-1|} (1) with the rectified linear unit defined as \operatorname{ReLU}(x)=max(x,0) and the absolute value function written as |x| . Similar to softmax, we need a numerically safe version to use in practice. We define numerically safe softpick as \operatorname{Softpick}(\mathbf{x})_{i}=\frac{\operatorname{ReLU}(e^{x_{i}-m}-e^{-m})}{\sum_{j=1}^{N}|e^{x_{j}-m}-e^{-m}|+\epsilon} (2) where m is the maximum value inside \mathbf{x} and 0<\epsilon\ll 1 is to avoid division by zero in the case that all inputs are exactly zero. This function is a drop-in replacement to softmax in the attention mechanism: \operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\operatorname{Softpick}\left(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{k}}}\right)\mathbf{V} (3) 3.2 Design Rationale To understand the reasoning behind each component of the softpick function, it is best we go through a step-by-step recreation of the formula. We start from the vanilla softmax function. \operatorname{Softmax}(\mathbf{x})_{i}=\frac{e^{x_{i}}}{\sum_{j=1}^{N}e^{x_{j}}} (4) The reason why we still do not modify this function too much is that the characteristics of softmax in training are ideal for attention. We want to maintain as much of the Jacobian matrix as possible and keep the gradient norm bounded that way. The next step would be to induce "null attention", or allowing attention to output zeros. We can achieve this by moving down the exponential by one, making it output negative values -1<y<0 for negative inputs. We then rectify this output with ReLU. \operatorname{Softmax}(\mathbf{x})_{i}=\frac{{\color[rgb]{1,0.5,0}\operatorname{ReLU}(}e^{x_{i}}{\color[rgb]{1,0.5,0}-1)}}{\sum_{j=1}^{N}{\color[rgb]{1,0.5,0}\operatorname{ReLU}(}e^{x_{j}}{\color[rgb]{1,0.5,0}-1)}} (5) This function has several desirable properties. First, it allows zero-valued outputs, which potentially enables sparsity optimizations and minimizes noise from the accumulation of unneeded values. Second, unlike softmax, the denominator allows for