The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention
*Giorgio Roffo · 2025
Read The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention by *Giorgio Roffo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14560.txt.
Description
Giorgio Roffo traces the roots of self-attention in a 2025 computing paper. Self-attention, the engine of Transformer models, decides how much each word or pixel should heed the others. Roffo argues the same trick powered an older method for picking useful features from data. Both rest on a table of pairwise affinities that steers how information flows.
The older method lets the affinity table come from expert knowledge or learning, then spreads scores across many hops. Self-attention builds its table fresh from the input and takes a single hop. Seen that way, attention is one member of a wider family. The history matters because the family's other members may solve attention's known weaknesses.
Catalog record
| CREATOR | *Giorgio Roffo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-14560