The Hidden Power of Normalization: Exponential Capacity Control in Deep Neural Networks
Khoat Than [email protected] · 2025
Read The Hidden Power of Normalization: Exponential Capacity Control in Deep Neural Networks by Khoat Than [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00958.txt.
Description
Khoat Than offers a theoretical explanation for why normalization layers, a standard part of deep neural networks, help training so much. These layers are known to steady training and improve generalization, but the mechanism, especially in very deep networks with many such layers, has stayed unclear.
Than frames the issue as capacity control. Without normalization, a network's Lipschitz constant, a measure of how sharply its output can change, can grow exponentially, and such over flexible networks are everywhere. Adding normalization layers cuts that constant at an exponential rate per layer. Two benefits follow: the loss landscape smooths, easing optimization, and the effective capacity shrinks, improving guarantees on unseen data.
Catalog record
| CREATOR | Khoat Than [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00958