Cascade Token Selection for Transformer Attention Acceleration

Stephen J. Thomas · 2026

Read Cascade Token Selection for Transformer Attention Acceleration by Stephen J. Thomas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03110.txt.

Description

Stephen J. Thomas speeds up representative token selection in transformer attention by reusing choices across layers. One attention method picks a small set of key tokens per layer, but building the full similarity matrix each time is costly.

The cascade inherits the prior layer's set, checks it with a smaller cross-matrix, and adds or drops a few tokens as needed. That cuts the selection cost sharply. On three model families, tests saved a large share of Gram work, and the key token sets overlapped strongly between layers. The result suggests informative tokens persist through network depth.

Catalog record

CREATORStephen J. Thomas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03110

Browse the full library · About the Tower of Babel