Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers

Shubham Aggarwal, Lokendra Kumar · 2026

Read Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers by Shubham Aggarwal, Lokendra Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08343.txt.

Description

Aggarwal and colleagues rethink attention output projection with structured Hadamard transforms for fast models. The dense output map in multi-head models scales fast with model size, adding heavily to parameter count, memory footprint, and runtime cost. The authors replace this map with a fixed, cost-free WHT followed by an affine map. This eliminates about 25 percent of block weights while keeping global cross-head mixing through a norm-preserving map. Transform models show steeper test loss curves per training FLOPs.

Catalog record

CREATORShubham Aggarwal, Lokendra Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23902
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08343

Browse the full library · About the Tower of Babel