Optimizing Mixture of Block Attention
Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, Song Han · 2025
Read Optimizing Mixture of Block Attention by Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, Song Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11571.txt.
Description
Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, and Song Han study Mixture of Block Attention. This design lets language models read long texts at low cost. The model attends to just a few blocks of stored keys and values. The idea is promising. But why it works, and how to run it fast, were both unclear.
A statistical model shows the key skill is the router's. The router must tell useful blocks from useless ones. A signal-to-noise ratio ties that skill to design choices. Guided by the analysis, the team improves the method. They also supply a fast GPU version. Long-context models stand to gain speed with no loss in quality.
Catalog record
| CREATOR | Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, Song Han |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11571