Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks
Yuval Ran-Milo · 2026
Read Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks by Yuval Ran-Milo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11487.txt.
Description
Ran-Milo proves that attention sinks are sometimes a must in softmax transformers, not a mere training quirk. A sink piles attention mass on one fixed, content-free spot. The proof uses a trigger task: on a set token, average all past token vectors; else output zero. Normalization over the probability simplex, he shows, forces attention onto a stable anchor for such default behavior. ReLU attention, which skips normalization, solves the same task with no sink at all. Experiments back the claim in single-head and multi-head models.
Catalog record
| CREATOR | Yuval Ran-Milo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74415 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11487