MonarchAttention : Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention

Can Yaras, Alec S. Xu, Pierre Abillama, Changwoo Lee, Laura Balzano · 2025

Read MonarchAttention : Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention by Can Yaras, Alec S. Xu, Pierre Abillama, Changwoo Lee, Laura Balzano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18698.txt.

Description

Can Yaras and co-authors present MonarchAttention, a method that speeds up the attention step inside Transformer models. Attention cost grows with the square of sequence length, which makes long inputs slow. Their method approximates softmax attention with Monarch matrices, a class of structured matrices that can express rich patterns at lower cost.

In their 2025 paper, an optimization method projects softmax attention onto that matrix class with no added training. Replacing every attention layer still costs little in task quality. Custom GPU kernels beat FlashAttention-2 in wall time, with the gain growing on longer sequences. Tests span vision and language tasks and several model designs, showing the swap holds up broadly.

Catalog record

CREATORCan Yaras, Alec S. Xu, Pierre Abillama, Changwoo Lee, Laura Balzano
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18698

Browse the full library · About the Tower of Babel