The Hidden Power of Normalization: Exponential Capacity Control in Deep Neural Networks

Khoat Than [email protected] · 2025

Read The Hidden Power of Normalization: Exponential Capacity Control in Deep Neural Networks by Khoat Than [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00958.txt.

Description

Khoat Than offers a theoretical explanation for why normalization layers, a standard part of deep neural networks, help training so much. These layers are known to steady training and improve generalization, but the mechanism, especially in very deep networks with many such layers, has stayed unclear.

Than frames the issue as capacity control. Without normalization, a network's Lipschitz constant, a measure of how sharply its output can change, can grow exponentially, and such over flexible networks are everywhere. Adding normalization layers cuts that constant at an exponential rate per layer. Two benefits follow: the loss landscape smooths, easing optimization, and the effective capacity shrinks, improving guarantees on unseen data.

Catalog record

CREATORKhoat Than [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00958

Browse the full library · About the Tower of Babel