Batch normalization does not improve initialization

Joris Dannemann, Gero Junike · 2025

Read Batch normalization does not improve initialization by Joris Dannemann, Gero Junike (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17913.txt.

Introduction

Joris Dannemann and Gero Junike dispute a claim about a standard neural network trick in a 2025 paper. The trick, batch normalization, rescales the outputs of each layer during training and is known to help learning. A well-known 2018 paper argued that part of its power comes from giving the network a better starting point, a claim about initialization.

The authors present a counterexample that shows the claim does not hold in general. In their setup, applying the rescaling at the start does not improve the start state in the way the earlier work claimed. The short paper narrows what can be said for why the method works. Better training effects must come from other parts of the training process, they argue.

Catalog record

CREATORJoris Dannemann, Gero Junike
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17913

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Batch normalization does not improve initialization Joris Dannemann, Gero Junike Abstract Batch normalization is one of the most important regularization techniques for neural networks, significantly improving training by centering the layers of the neural network. There have been several attempts to provide a theoretical justification for batch normalization. Santurkar and Tsipras (2018) [How does batch normalization help optimization? Advances in neural information processing systems, 31] claim that batch normalization improves initialization. We provide a counterexample showing that this claim is not true, i.e., batch normalization does not improve initialization. Keywords: neural network, batch normalization, initialization 1 Introduction Neural networks are functions with many parameters. To train a neural network, the parameters are randomly initialized at the beginning of the training and are then calibrated to a learning set by minimizing a loss function via gradient decent, see [ 4 ] . Batch normalization (BN), see [ 6 ] , is a technique to improve the training of neural networks. It has been used in many applications [ 6 , 4 , 10 , 11 , 2 , 7 ] to make training faster and more stable. According to [ 4 , p. 317] , “Batch normalization (Ioffe and Szegedy, 2015) is one of the most exciting recent innovations in optimizing deep neural networks”. Similarly, [ 1 , p. 1] concludes: “Nowadays, there is little disagreement in the machine learning community that BN accelerates training, enables higher learning rates, and improves generalization accuracy […] and BN has successfully proliferated throughout all areas of deep learning”. However, the working mechanism of BN is not yet fully understood. The authors in [ 8 ] are among the first to analyze BN from a more theoretical point of view. They provide several theoretical arguments why BN improves training, in particular they claim in Lemma 4.5 “that BN also offers an advantage in initialization”, see [ 8 , p. 8] . This statement, i.e., [ 8 , Lemma 4.5] , is also cited in [ 11 , Lemma 7.2] . However, we provide a straightforward counterexample showing that this lemma is incorrect, i.e., BN does not improve initialization. This article is structured as follows: in Section 2 we give a brief introduction to neural networks and BN. In Section 3 , we provide a counterexample showing that BN does not improve initialization. 2 Neural networks and batch normalization For an introduction to neural networks with BN, see [ 4 , Section 8.7.1] and [ 11 , Section 7.15] . In this section, we summarize the most important definitions: A neural network is a function from some input space (e.g., a subspace of \mathbb{R}^{p} ) to some output space (e.g., a subspace of \mathbb{R} ). A neural network consists of layers, that consist of neurons , which use activation functions . The regularization technique batch normalization acts on one or more layers. We write \|\cdot\| and \langle\cdot,\cdot\rangle for the Euclidean norm and the inner product on \mathbb{R}^{d} , respectively. Definition 2.1 . An activation function is a function g:\mathbb{R}\to\mathbb{R} . For example, the identity, Sigmoid, tanh and ReLu (Rectified linear unit) activation functions are defined by x\mapsto x , x\mapsto\frac{1}{1+e^{-x}} , x\mapsto\tanh(x) and x\mapsto\max(x,0) , respectively. Next, we define a neuron, that has k inputs and one output. Definition 2.2 . A neuron is a function h with weights \boldsymbol{w}\in\mathbb{R}^{k} and bias b\in\mathbb{R} and activation function g , such that \displaystyle h: \displaystyle\mathbb{R}^{k}\to\mathbb{R},\quad\boldsymbol{x}\mapsto g\big(\langle\boldsymbol{w},\boldsymbol{x}\rangle+b\big). Definition 2.3 . Let h_{1},...,h_{n}:\mathbb{R}^{k}\to\mathbb{R} be n neurons. A layer is a function f:\,\mathbb{R}^{k}\to\mathbb{R}^{n},\quad\boldsymbol{x}\mapsto\big(h_{1}(\boldsymbol{x}),...,h_{n}(\boldsymbol{x})\big)^{\intercal} consisting of n\in\mathbb{N} neurons. Each neuron has k inputs. Example 2.4 . In the special case that each neuron of the layer has the identity as its activation function and zero bias, it holds that f(\boldsymbol{x})=W\boldsymbol{x} for a matrix W\in\mathbb{R}^{n\times k} of weights. Definition 2.5 . Suppose the input space has dimension p . A neural network (NN) consists of L\in\mathbb{N} layers f_{1}:\mathbb{R}^{k_{1}}\to\mathbb{R}^{n_{1}} ,…, f_{L}:\mathbb{R}^{k_{L}}\to\mathbb{R}^{n_{L}} with k_{1}=p , k_{2}=n_{1} ,…, k_{L}=n_{L-1} . The layers have n_{1},n_{2},...,n_{L} neurons. The neural network is the function \varphi_{\Theta}:\mathbb{R}^{p}\to\mathbb{R}^{n_{L}},\quad\boldsymbol{x}\mapsto f_{L}\big(f_{L-1}\big(...f_{2}\big(f_{1}(\boldsymbol{x})\big). The input \boldsymbol{x} is also called the input layer . The last layer f_{L} is also called the output layer . The layers f_{1},...,f_{L-1} are called hidden layers. All parameters of the NN, e.g., the weights and the biases of the neurons, are stored in a vector \Theta . The optimal parameters \Theta^{\ast} of the NN are usually found via minimizing some loss function, e.g., \Theta^{\ast}=\text{argmin}_{\Theta}\frac{1}{N}\sum_{i=1}^{N}\bigg(\varphi_{\Theta}\big(\bar{\boldsymbol{x}}^{(i)}\big)-\bar{y}^{(i)}\bigg)^{2}, where \bar{\boldsymbol{x}}^{(i)}\in\mathbb{R}^{p} and \bar{y}^{(i)}\in\mathbb{R} and \big\{\big(\bar{\boldsymbol{x}}^{(1)},\bar{y}^{(1)}\big),...,\big(\bar{\boldsymbol{x}}^{(N)},\bar{y}^{(N)}\big)\big\} is a learning set of size N . The loss function is minimized by (stochastic) gradient descent. It is recommended to initialize \Theta randomly, see [ 3 , 5 ] . During the optimization, the learning set is divided into several disjoint sets, so called batches . Next, we define layers with BN, introduced by [ 6 ] . Note that the input layer of a batch of size M has dimension p\times M and the output of the l -th layer has dimension n_{l}\times M . Definition 2.6 . Consider a NN with of L layers. The l -th layer f_{l}:\mathbb{R}^{k_{l}}\to\mathbb{R}^{n_{l}} consists of n_{l} neurons. Apply the NN to a batch of size M\in\mathbb{N} . The output of the l -th layer has the form \mathbb{\mathbb{Z}}_{l}=\left(\begin{array}[]{ccc}z_{11}&...&z_{1M}\\ \vdots&&\vdots\\ z_{n_{l}1}&...&z_{n_{l}M}\end{array}\right)\in\mathbb{R}^{n_{l}\times M}. Let \mu_{j}=\frac{1}{M}\sum_{m=1}^{M}z_{jm},\quad\text{and $\quad\sigma_{j}=\sqrt{\frac{1}{M}\sum_{m=1}^{M}(z_{jm}-\mu_{j})^{2}}$},\quad j=1,...,n_{l}. Let \gamma_{1},...,\gamma_{n_{l}}\in\mathbb{R} and \beta_{1},...,\beta_{n_{l}}\in\mathbb{R} be parameters and define \hat{\mathbb{\mathbb{Z}}}_{l}=\left(\begin{array}[]{ccc}\gamma_{1}\frac{z_{11}-\mu_{1}}{\sigma_{1}}+\beta_{1}&...&\gamma_{1}\frac{z_{1M}-\mu_{1}}{\sigma_{1}}+\beta_{1}\\ \vdots&&\vdots\\ \gamma_{n_{l}}\frac{z_{n_{l}1}-\mu_{n_{l}}}{\sigma_{n_{l}}}+\beta_{n_{l}}&...&\gamma_{n_{l}}\frac{z_{n_{l}M}-\mu_{n_{l}}}{\sigma_{n_{l}}}+\beta_{n_{l}}\end{array}\right)\in\mathbb{R}^{n_{l}\times M}. \hat{\mathbb{\mathbb{Z}}}_{l} is called batch normalization transform of layer \mathbb{\mathbb{Z}}_{l} . Remark 2.7 . During the training, the NN is trained on the transformed data \hat{\mathbb{\mathbb{Z}}}_{l} and the parameters \gamma_{j} and \beta_{j} , j=1,...,n_{l} are trainable as well. If \gamma_{j}=\sigma_{j} and \beta_{j}=\frac{\mu_{j}}{\sigma_{j}} then \hat{\mathbb{\mathbb{Z}}}_{l}=\mathbb{Z}_{l} , i.e., one “could recover the original activations, if that were the optimal thing to do”, see [ 6 , Section 3, p. 3] . For prediction, the average values (over all batches) of the parameters \mu_{j} and \sigma_{j} are used. 3 Counterexample: BN does not improve initialization [ 8 ] analyze individual layers of a NN assuming that all neurons in that layer have the identity as activation function and that \beta and \gamma for the BN are constants. In [ 8 , Lemma 4.5] it is claimed that BN offers an advantage in initialization: “(BatchNorm leads to a favorable initialization). Let \boldsymbol{W}^{\boldsymbol{\ast}} and \widehat{\boldsymbol{W}}^{\boldsymbol{*}} be the set of local optima for the weights in the normal and BN networks, respectively. For any initialization W_{0} \left\|W_{0}-\widehat{W}^{*}\right\|^{2}\leq\|W_{0}-W^{*}\|^{2}-\frac{1}{\|W^{*}\|^{2}}\left(\|W^{*}\|^{2}-\langle W^{*},W_{0}\rangle\right)^{2}, (1) if \langle W_{0},W^{*}\rangle>0 , where \widehat{W}^{*} and W^{*} are closest optima for BN and standard network, respectively.” , see [ 8 , p. 9] . Remark 3.1 . The matrix W_{0}\in\mathbb{R}^{n\times k} is understood as the initial weights of a particular layer with k inputs consisting of n neurons with biases equal to zero. W^{*} and \widehat{W}^{*} are the locally optimal weights of the layer with respect to the loss function with and without BN, respectively. The norm and the inner product applied to matrices in Inequality ( 1 ) are understood column wise. In the next example, we will show that [ 8 , Lemma 4.5] is wrong. We will construct a NN consisting of a single neuron with the identity as activation function with initial weights W_{0} identical to the optimal weights W^{\ast} and show that any weights \widehat{W}^{*} satisfying Inequality

Read the complete text