Poly-attention: a general scheme for higher-order self-attention

Sayak Chakrabarti, Toniann Pitassi, Josh Alman · 2026

Read Poly-attention: a general scheme for higher-order self-attention by Sayak Chakrabarti, Toniann Pitassi, Josh Alman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02422.txt.

Description

Sayak Chakrabarti and 2 co-authors define a broad family of attention mechanisms that can notice relationships among three or more tokens at once, in a 2026 arXiv paper. Standard self-attention, the core of transformer models, handles pairs well but fails at basic tasks needing triples or composition. Earlier higher-order alternatives could do some of those tasks only with slow, superquadratic running times.

The new class, poly-attention, allows arbitrary higher-order computations and arbitrary relationship structures between tokens, with older proposals as special cases. The authors map its computational complexity and representational strength, proving matching upper and lower bounds for exact and approximate computation. One new mechanism runs in exactly quadratic time yet composes any fixed number of functions. Earlier mechanisms could not compose even two functions that fast, and the lower bounds show they never will.

Catalog record

CREATORSayak Chakrabarti, Toniann Pitassi, Josh Alman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE197 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02422

Browse the full library · About the Tower of Babel