CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers

Yoshihiro Yamada · 2025

Read CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers by Yoshihiro Yamada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06704.txt.

Description

Yoshihiro Yamada, a solo author, rewrites the attention step of transformer networks as a circular convolution to cut its cost, in a 2025 paper. Attention is the engine of modern language and vision models, but its cost grows with the square of the sequence length, blocking long inputs. The proposed method, CAT, works in the frequency domain.

By applying convolutions through Fourier transforms, CAT drops the cost to a near-linear rate. It also trims the heavy layers around attention, so the model carries fewer parameters and adds no costly side operations. Early tests in a plain framework show steady accuracy gains and about a tenth faster running. Faster attention at the same quality stretches every training and serving budget.

Catalog record

CREATORYoshihiro Yamada
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06704

Browse the full library · About the Tower of Babel