Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation
Franz A. Heinsen, Leo Kozachkov · 2026
Read Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation by Franz A. Heinsen, Leo Kozachkov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00294.txt.
Description
Franz A. Heinsen and Leo Kozachkov show self attention can run at constant cost per token via symmetry aware Taylor approximation. Standard self attention costs grow with context length, driving storage, compute, and energy demand beyond easy supply. They decompose the Taylor expansion into expressions built on symmetric chains of tensor products. Exploiting that symmetry yields feed forward maps from queries and keys to a minimal polynomial kernel feature basis, with cost fixed inversely to head size.
Catalog record
| CREATOR | Franz A. Heinsen, Leo Kozachkov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36657 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.00294