MonarchAttention : Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention
Can Yaras, Alec S. Xu, Pierre Abillama, Changwoo Lee, Laura Balzano · 2025
Read MonarchAttention : Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention by Can Yaras, Alec S. Xu, Pierre Abillama, Changwoo Lee, Laura Balzano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18698.txt.
Description
Can Yaras and co-authors present MonarchAttention, a method that speeds up the attention step inside Transformer models. Attention cost grows with the square of sequence length, which makes long inputs slow. Their method approximates softmax attention with Monarch matrices, a class of structured matrices that can express rich patterns at lower cost.
In their 2025 paper, an optimization method projects softmax attention onto that matrix class with no added training. Replacing every attention layer still costs little in task quality. Custom GPU kernels beat FlashAttention-2 in wall time, with the gain growing on longer sequences. Tests span vision and language tasks and several model designs, showing the swap holds up broadly.
Catalog record
| CREATOR | Can Yaras, Alec S. Xu, Pierre Abillama, Changwoo Lee, Laura Balzano |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18698