Adaptive Head Budgeting for Efficient Multi-Head Attention

Bilal FAYE, Abdoulaye MBAYE, Hanane AZZAG, Mustapha Lebbahe-mail: [email protected], [email protected], [email protected], [email protected] · 2026

Read Adaptive Head Budgeting for Efficient Multi-Head Attention by Bilal FAYE, Abdoulaye MBAYE, Hanane AZZAG, Mustapha Lebbahe-mail: [email protected], [email protected], [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22583.txt.

Description

Faye and three colleagues present BudgetFormer, a Transformer that spends attention heads only where they pay off. Multi-head attention lets Transformers capture varied codes, but every head usually fires for every input. For coarse tasks such as text classification, where key information is global, fixed allocation wastes computation. BudgetFormer instead learns a per-input head budget and relevance distribution to pick the most informative heads, balancing search and use in training. On text classification tasks it cuts computation and memory use while matching or beating standard multi-head attention.

Catalog record

CREATORBilal FAYE, Abdoulaye MBAYE, Hanane AZZAG, Mustapha Lebbahe-mail: [email protected], [email protected], [email protected], [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44886
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22583

Browse the full library · About the Tower of Babel