Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers
Shubham Aggarwal, Lokendra Kumar · 2026
Read Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers by Shubham Aggarwal, Lokendra Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08343.txt.
Description
Aggarwal and colleagues rethink attention output projection with structured Hadamard transforms for fast models. The dense output map in multi-head models scales fast with model size, adding heavily to parameter count, memory footprint, and runtime cost. The authors replace this map with a fixed, cost-free WHT followed by an affine map. This eliminates about 25 percent of block weights while keeping global cross-head mixing through a norm-preserving map. Transform models show steeper test loss curves per training FLOPs.
Catalog record
| CREATOR | Shubham Aggarwal, Lokendra Kumar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23902 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08343