Gating Enables Curvature: A Geometric Expressivity Gap in Attention
Satwik Bathula, Anand A. Joshi · 2026
Read Gating Enables Curvature: A Geometric Expressivity Gap in Attention by Satwik Bathula, Anand A. Joshi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14702.txt.
Description
Bathula and Joshi explain gated attention with geometry. They treat attention outputs as Gaussian means and study their Fisher-Rao geometry. Plain attention, with its affine structure, stays confined to flat statistical manifolds by nature. Gating opens up curved ones, including positively curved manifolds the ungated form can never reach. Tests matched the theory: gated models curved more and did better where dividing lines bend, with no steady edge where lines stay straight. The curve also grew steadily with depth.
Catalog record
| CREATOR | Satwik Bathula, Anand A. Joshi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85234 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14702