Poly-attention: a general scheme for higher-order self-attention
Sayak Chakrabarti, Toniann Pitassi, Josh Alman · 2026
Read Poly-attention: a general scheme for higher-order self-attention by Sayak Chakrabarti, Toniann Pitassi, Josh Alman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02422.txt.
Description
Sayak Chakrabarti and 2 co-authors define a broad family of attention mechanisms that can notice relationships among three or more tokens at once, in a 2026 arXiv paper. Standard self-attention, the core of transformer models, handles pairs well but fails at basic tasks needing triples or composition. Earlier higher-order alternatives could do some of those tasks only with slow, superquadratic running times.
The new class, poly-attention, allows arbitrary higher-order computations and arbitrary relationship structures between tokens, with older proposals as special cases. The authors map its computational complexity and representational strength, proving matching upper and lower bounds for exact and approximate computation. One new mechanism runs in exactly quadratic time yet composes any fixed number of functions. Earlier mechanisms could not compose even two functions that fast, and the lower bounds show they never will.
Catalog record
| CREATOR | Sayak Chakrabarti, Toniann Pitassi, Josh Alman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 197 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02422