CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers
Yoshihiro Yamada · 2025
Read CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers by Yoshihiro Yamada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06704.txt.
Description
Yoshihiro Yamada, a solo author, rewrites the attention step of transformer networks as a circular convolution to cut its cost, in a 2025 paper. Attention is the engine of modern language and vision models, but its cost grows with the square of the sequence length, blocking long inputs. The proposed method, CAT, works in the frequency domain.
By applying convolutions through Fourier transforms, CAT drops the cost to a near-linear rate. It also trims the heavy layers around attention, so the model carries fewer parameters and adds no costly side operations. Early tests in a plain framework show steady accuracy gains and about a tenth faster running. Faster attention at the same quality stretches every training and serving budget.
Catalog record
| CREATOR | Yoshihiro Yamada |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06704