Learning in Compact Spaces with Approximately Normalized Transformer
Jörg K.H. Franke, Urs Spiegelhalter, Marianna Nezhurina, Jenia Jitsev, Frank Hutter, Michael Hefenbrock · 2025
Read Learning in Compact Spaces with Approximately Normalized Transformer by Jörg K.H. Franke, Urs Spiegelhalter, Marianna Nezhurina, Jenia Jitsev, Frank Hutter, Michael Hefenbrock (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22014.txt.
Description
Franke and co-authors trained transformers on a sphere in a 2025 paper. Deep nets fight overfitting, blow-ups, and a residual stream whose spread grows with depth. The classic fixes, weight penalties and normalization layers, add knobs that need tuning.
Forcing all weights and codes onto a sphere kills those knobs but costs dear. Their near-normalized transformer instead uses plain scalar rescales, steered by a tightness idea from physics. The result keeps most of the sphere's gains at a sliver of the cost. Faster, steadier training is the prize for large model builders.
Catalog record
| CREATOR | Jörg K.H. Franke, Urs Spiegelhalter, Marianna Nezhurina, Jenia Jitsev, Frank Hutter, Michael Hefenbrock |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22014