Just One Layer Norm Guarantees Stable Extrapolation
Juliusz Ziomek, George Whittle ⋆, Michael A. OsborneMachine Learning Research Group, University of Oxford ⋆ Equal Contribution † Corresponding Author {juliusz, george, mosb} @ robots.ox.ac.uk · 2025
Read Just One Layer Norm Guarantees Stable Extrapolation by Juliusz Ziomek, George Whittle ⋆, Michael A. OsborneMachine Learning Research Group, University of Oxford ⋆ Equal Contribution † Corresponding Author {juliusz, george, mosb} @ robots.ox.ac.uk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14512.txt.
Introduction
Juliusz Ziomek, George Whittle, and Michael Osborne prove a stability result in twenty twenty-five. Neural networks behave unpredictably far from their training data. Extrapolation results existed only for specific cases. The authors give general theorems, the first of their kind, using neural tangent kernel theory.
They analyze infinitely wide networks trained to convergence. Adding just one layer normalization layer changes everything, they prove. The induced kernel becomes one of bounded variance. The network's output then stays bounded even on inputs far outside the training range. Without normalization, a broad class of networks can produce pathologically huge outputs. Experiments on finite networks back the theory. Readers get a simple architectural guarantee against runaway extrapolation.
Catalog record
| CREATOR | Juliusz Ziomek, George Whittle ⋆, Michael A. OsborneMachine Learning Research Group, University of Oxford ⋆ Equal Contribution † Corresponding Author {juliusz, george, mosb} @ robots.ox.ac.uk |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 143 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14512
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Just One Layer Norm Guarantees Stable Extrapolation Juliusz Ziomek, George Whittle ⋆, Michael A. OsborneMachine Learning Research Group, University of Oxford ⋆ Equal Contribution † Corresponding Author {juliusz, george, mosb} @ robots.ox.ac.uk Abstract In spite of their prevalence, the behaviour of Neural Networks when extrapolating far from the training distribution remains poorly understood, with existing results limited to specific cases. In this work, we prove general results—the first of their kind—by applying Neural Tangent Kernel (NTK) theory to analyse infinitely-wide neural networks trained until convergence and prove that the inclusion of just one Layer Norm (LN) fundamentally alters the induced NTK, transforming it into a bounded-variance kernel. As a result, the output of an infinitely wide network with at least one LN remains bounded, even on inputs far from the training data. In contrast, we show that a broad class of networks without LN can produce pathologically large outputs for certain inputs. We support these theoretical findings with empirical experiments on finite-width networks, demonstrating that while standard NNs often exhibit uncontrolled growth outside the training domain, a single LN layer effectively mitigates this instability. Finally, we explore real-world implications of this extrapolatory stability, including applications to predicting residue sizes in proteins larger than those seen during training and estimating age from facial images of underrepresented ethnicities absent from the training set. 1 Introduction Neural Networks (NN) have dominated the Machine Learning landscape for more than a decade. Various deep architectures became the state-of-the-art approaches across countless applications. However, despite this empirical success, we still have a limited understanding of how these models learn and how they behave when making predictions. What poses particular difficulty is analysing the network’s prediction as inputs get further away from the training data. Clearly, a network achieving a low training set error should output values close to the ground truth labels on the training points. Yet, it is difficult to say how the network will behave outside of the training domain, as the training process does not necessarily constrains the network’s output there. As such, the extrapolatory behaviour of these networks is very much undefined. Lack of understanding of this extrapolatory behaviour can be dangerous. For example, a deep learning-based control system that applies voltages to a robot arm based on camera inputs can encounter pixel values with brightness exceeding those found in the training set. If this causes the network to output extreme values, this could easily lead to a serious safety hazard in the system. As such, understanding how networks behave outside of the training domain and whether their predictions are stable out of distribution is critical. One of the most complete theoretical approaches to studying the behaviour of Deep Neural Networks is the Neural Tangent Kernel (NTK) theory [ 19 , 23 ] . Previous work in that stream [ 23 ] managed to show that as the network width approaches infinity, the trained network’s output is equivalent to the posterior mean of a Gaussian Process (GP) [ 29 ] with a specific kernel function that depends only on the network’s architecture. While any real neural network has, of course, a finite width, in many cases, the predictions made by the NTK theory matched the empirical observations [ 19 , 23 , 1 , 13 ] . In this work, we use NTK theory to analyse the effect on extrapolatory, or out-of-distribution (OOD), predictions of including Layer Normalisation (LN) [ 2 ] operations in a network. We show that for an infinitely-wide, fully-connected standard network with positive n -homogeneous nonlinearities, there always exists a fixed training dataset such that their predictions get pathologically large, i.e. “explode”. On the other hand, we show that the NTK of infinitely-wide, fully-connected architectures containing at least one LN operation admits an upper bound for any input, no matter how far from the training domain. Using this, we show that such networks trained until convergence have a bounded output even infinitely far from the training domain, for any training set. We then show how these properties are useful across different practical applications, including extrapolating residue size prediction to proteins larger than those seen during training and age prediction for unrepresented ethnicities absent from training data. We detail our contributions below: • We show with Theorem 3.1 that for infinitely-wide fully-connected positive n -homogeneous nonlinearities, but without any LN, the absolute value of the expectation (with expectation taken over initialisation) can be arbitrarily high outside of the training domain. • We show with Theorem 3.3 that the inclusion of a single LN anywhere in the network imposes an upper bound on the absolute value of expected output anywhere in the domain. That is, they enjoy stable extrapolation. • We verify these theoretical results on a number of toy problems and show that standard neural networks “explode” outside of the training domain, whereas the presence of at least one LN prevents this explosion from happening. • We show how these properties are useful when making out-of-distribution predictions, such as extrapolating residue size prediction to proteins larger than those seen during training and age prediction for unrepresented ethnicities absent from training data. Furthermore, we extend our extrapolation stability results with Theorem C.3 to a more general class of nonlinearities and with Theorem C.1 to, under certain circumstances, any architecture within the Tensor Program framework [ 35 , 36 , 37 ] . 2 Preliminaries and Problem Setting We assume we are solving a supervised learning problem given a dataset \mathcal{D}_{\textrm{train}}=\{(\bm{x}_{i},y_{i})\}_{i=1}^{\theta} , where for all i=1,\dots,n , by \bm{x}_{i}\in\mathbb{R}^{n_{0}} we denote an n_{0} -dimensional input datapoint and by y_{i}\in\mathbb{R} corresponding to the target variable. We assume there are no repeated datapoints, that is, \bm{x}_{i}=\bm{x}_{j}\implies i=j . We will denote by \mathcal{X}_{\textrm{train}}=\{\bm{x}_{i}\}_{i=1}^{n} and \mathcal{Y}_{\textrm{train}}=\{y_{i}\}_{i=1}^{n} . We use this data to train a NN f_{\theta}:\mathbb{R}^{n_{0}}\to\mathbb{R} , parametrised by some \theta\in\Omega\subset\mathbb{R}^{d} . We measure the success on this learning problem by the MSE loss function L(\theta;\mathcal{D}_{\textrm{train}})=\frac{1}{2}\left\lVert f_{\theta}(\bm{x}_{i})-y_{i}\right\lVert_{2}^{2} . We assume the parameters of NN are initialised as described in Appendix A . They then evolve via gradient descent update as \theta^{t}=\theta^{t-1}-\eta\nabla_{\theta}L(\theta;\mathcal{D}_{\textrm{train}}) with learning rate \eta>0 at each step t>0 . The work of [ 19 ] observed that (even finite) NNs during training follow a kernel gradient descent with a kernel: \Theta(\bm{x},\bm{x}^{\prime};\theta)=\langle\nabla_{\theta}f_{\theta}(\bm{x}),\nabla_{\theta}f_{\theta}(\bm{x}^{\prime})\rangle, also known as the empirical NTK . They then show that for an MLP, as the width of the smallest layer approaches infinity, the NTK converges to a limit \Theta(\bm{x},\bm{x}^{\prime}) that stays constant throughout the training process. Furthermore, the work of [ 23 ] showed that if the network is trained with learning rate small enough then in the limit t\to\infty , the network converges to: f_{\theta^{\infty}}(\bm{x}^{\star})=\Theta(\bm{x}^{\star},\mathcal{X}_{\textrm{train}})\Theta(\mathcal{X}_{\textrm{train}},\mathcal{X}_{\textrm{train}})^{-1}(\mathcal{Y}_{\textrm{train}}-f_{\theta^{0}}(\bm{x}^{\star}))+f_{\theta^{0}}(\bm{x}^{\star}), where f_{\theta^{0}}(\bm{x}^{\star}) is the output of freshly initalised network with parameters \theta^{0} for input \bm{x}^{\star} . As for the standard parameterisation \mathbb{E}[f_{\theta^{0}}(\bm{x}^{\star})] = 0, we thus also get that: \mathbb{E}[f_{\theta^{\infty}}(\bm{x}^{\star})]=\Theta(\bm{x}^{\star},\mathcal{X}_{\textrm{train}})\Theta(\mathcal{X}_{\textrm{train}},\mathcal{X}_{\textrm{train}})^{-1}\mathcal{Y}_{\textrm{train}}, which is precisely the posterior mean of a GP with kernel function \Theta(\bm{x},\bm{x}^{\prime}) conditioned on data \mathcal{D}_{\textrm{train}} evaluated at point \bm{x}^{\star} . While the original result of [ 23 ] assumed \left\lVert\bm{x}^{\star}\right\rVert\leq 1 , this assumption can be lifted without affecting the limit, as we explain in Appendix B . This result is valid, as long as the infinite width limit \Theta(\bm{x},\bm{x}^{\prime}) exists. The work