Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks

Yuval Ran-Milo · 2026

Read Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks by Yuval Ran-Milo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11487.txt.

Description

Ran-Milo proves that attention sinks are sometimes a must in softmax transformers, not a mere training quirk. A sink piles attention mass on one fixed, content-free spot. The proof uses a trigger task: on a set token, average all past token vectors; else output zero. Normalization over the probability simplex, he shows, forces attention onto a stable anchor for such default behavior. ReLU attention, which skips normalization, solves the same task with no sink at all. Experiments back the claim in single-head and multi-head models.

Catalog record

CREATORYuval Ran-Milo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74415
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11487

Browse the full library · About the Tower of Babel