Optimizing Mixture of Block Attention

Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, Song Han · 2025

Read Optimizing Mixture of Block Attention by Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, Song Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11571.txt.

Description

Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, and Song Han study Mixture of Block Attention. This design lets language models read long texts at low cost. The model attends to just a few blocks of stored keys and values. The idea is promising. But why it works, and how to run it fast, were both unclear.

A statistical model shows the key skill is the router's. The router must tell useful blocks from useless ones. A signal-to-noise ratio ties that skill to design choices. Guided by the analysis, the team improves the method. They also supply a fast GPU version. Long-context models stand to gain speed with no loss in quality.

Catalog record

CREATORGuangxuan Xiao, Junxian Guo, Kasra Mazaheri, Song Han
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11571

Browse the full library · About the Tower of Babel