Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

Franz A. Heinsen, Leo Kozachkov · 2026

Read Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation by Franz A. Heinsen, Leo Kozachkov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00294.txt.

Description

Franz A. Heinsen and Leo Kozachkov show self attention can run at constant cost per token via symmetry aware Taylor approximation. Standard self attention costs grow with context length, driving storage, compute, and energy demand beyond easy supply. They decompose the Taylor expansion into expressions built on symmetric chains of tensor products. Exploiting that symmetry yields feed forward maps from queries and keys to a minimal polynomial kernel feature basis, with cost fixed inversely to head size.

Catalog record

CREATORFranz A. Heinsen, Leo Kozachkov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36657
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.00294

Browse the full library · About the Tower of Babel