Gating Enables Curvature: A Geometric Expressivity Gap in Attention

Satwik Bathula, Anand A. Joshi · 2026

Read Gating Enables Curvature: A Geometric Expressivity Gap in Attention by Satwik Bathula, Anand A. Joshi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14702.txt.

Description

Bathula and Joshi explain gated attention with geometry. They treat attention outputs as Gaussian means and study their Fisher-Rao geometry. Plain attention, with its affine structure, stays confined to flat statistical manifolds by nature. Gating opens up curved ones, including positively curved manifolds the ungated form can never reach. Tests matched the theory: gated models curved more and did better where dividing lines bend, with no steady edge where lines stay straight. The curve also grew steadily with depth.

Catalog record

CREATORSatwik Bathula, Anand A. Joshi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85234
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14702

Browse the full library · About the Tower of Babel