Quantitative convergence of trained single layer neural networks to Gaussian processes

Eloy Mosig García, Andrea Agazzi, Dario Trevisan · 2025

Read Quantitative convergence of trained single layer neural networks to Gaussian processes by Eloy Mosig García, Andrea Agazzi, Dario Trevisan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24544.txt.

Introduction

Wide neural networks behave like Gaussian processes, a fact often stated without error bars. Eloy Mosig and 2 co-authors quantify that convergence for shallow networks trained by gradient descent. They bound the quadratic Wasserstein distance between the network output and its Gaussian approximation at any training time, including times well into training, where prior estimates were scarce.

The distance decays as a power of the network width. The bounds show how width and input dimension shape the error, and how training dynamics move it over time. Finite-width estimates like these tell practitioners when the Gaussian picture can be trusted numerically.

Catalog record

CREATOREloy Mosig García, Andrea Agazzi, Dario Trevisan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE137 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24544

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantitative convergence of trained single layer neural networks to Gaussian processes Eloy Mosig García, Andrea Agazzi, Dario Trevisan Abstract In this paper, we study the quantitative convergence of shallow neural networks trained via gradient descent to their associated Gaussian processes in the infinite-width limit. While previous work has established qualitative convergence under broad settings, precise, finite-width estimates remain limited, particularly during training. We provide explicit upper bounds on the quadratic Wasserstein distance between the network output and its Gaussian approximation at any training time t ≥ 0 t\geq 0 , demonstrating polynomial decay with network width. Our results quantify how architectural parameters, such as width and input dimension, influence convergence, and how training dynamics affect the approximation error. 1 Introduction Deep neural networks have achieved remarkable success across a wide range of tasks in computer vision, natural language processing, and scientific computing, often surpassing traditional models by large margins LeCun et al. 2015 , Goodfellow et al. 2016 . This empirical progress has sparked substantial interest in understanding the theoretical principles underlying their behavior, particularly in the overparameterized regime, where the number of parameters is larger than the one of training samples. A major avenue of theoretical investigation in this sense focuses on studying the properties of neural networks in the infinite-width limit. For instance, when the network’s parameters at initialization are sampled from a Gaussian distribution, it was shown Neal 1996 , de G. Matthews et al. 2018 that the network’s output converges to a Gaussian process as width tends to infinity, providing a tractable framework for theoretical analysis. This perspective was significantly extended by the introduction of the Neural Tangent Kernel (NTK) framework Jacot et al. 2018 , allowing to characterize the training dynamics of infinitely wide neural networks under gradient descent in function space. In this limit, the network evolves approximately linearly around its initialization, and training can be understood as kernel regression with a fixed kernel which depends on the the architecture only, and is evaluated on input data. This linearization dramatically simplifies the analysis of generalization and convergence, and has led to a large body of theoretical work on the expressivity and limitations of infinitely wide networks. However, the practical relevance of NTK-based analyses hinges on the accuracy of their approximation at finite width. While existing literature has established qualitative convergence of wide neural networks in the NTK regime to Gaussian processes at positive training time Lee et al. 2020 , rigorous quantitative results — providing explicit finite-width error bounds — remain scarce. This gap limits the applicability of NTK theory to realistic settings where network width is large but finite. Indeed, quantitative convergence guarantees are crucial to bridge theory and practice. They allow one to bound the discrepancy between the predictions of a finite-width network and its infinite-width NTK counterpart, thus enabling quantitative uncertainty quantification estimates and the safe deployment of theoretical insights to real-world architectures. Moreover, such estimates reveal how network width, depth, initialization, and training hyperparameters impact the validity of linear approximations during training. These insights are essential for developing principled training strategies and for diagnosing when the NTK regime offers a reliable approximation, or when nonlinear effects beyond NTK must be taken into account. 1.1 Our contributions This paper provides rigorous quantitative estimates for the convergence of trained shallow neural networks towards their Gaussian process counterparts, measured in terms of quadratic Wasserstein distances. Specifically, we extend previously established convergence bounds at initialization obtained by Basteri and Trevisan 2024 , Favaro et al. 2025 and Trevisan 2023 to arbitrary positive training times. Our results deliver explicit convergence rates that decay polynomially with network width, clearly delineating how the approximation error evolves during training. Concretely, we demonstrate that the distance of the distribution of a shallow neural network’s output trained via gradient descent to its Gaussian process approximation at any training time t>0 satisfies explicit quantitative bounds dependent on network width. Our main theorem ( 3.4 ) shows that for any test point x , under mild assumptions on the hidden layer width n_{1} and on the regularity of the activation function we have: \mathcal{W}_{2}^{2}(f_{t}(x),G_{t}(x))=\mathcal{O}\left(\frac{\log n_{1}}{n_{1}}\right). We also address long-term training dynamics explicitly, characterizing convergence rates as training time diverges. Indeed, the above result continues to hold on timescales t growing polynomially in the network width n_{1} , as discussed in Remark 3.5 . These results significantly refine prior qualitative statements, providing actionable quantitative guidelines on how network parameters and training duration determine the extent to which finite networks emulate their infinite-width limits. 1.2 Related work The convergence of randomly initialized neural networks to Gaussian processes in the infinite-width limit has been a foundational result in the theory of neural networks. This phenomenon was first suggested by Neal 1996 and later formalized for deep architectures by de G. Matthews et al. 2018 . The key insight that this correspondence extends beyond initialization was introduced by Jacot et al. 2018 , who demonstrated that training dynamics in the infinite-width limit are governed by the so-called Neural Tangent Kernel (NTK), a deterministic kernel that linearizes the training trajectory. This sparked significant interest in the use of kernel methods to analyze deep learning models. Following these developments, several works studied the convergence of finite-width neural networks to their limiting Gaussian processes. In particular, Lee et al. 2020 established that gradient descent dynamics in the NTK regime converge to those of a linearized model. More recently, the works of Basteri and Trevisan 2024 and Trevisan 2023 provided quantitative convergence rates at initialization, measured in Wasserstein distance, which laid the groundwork for a more refined understanding of the finite-width behavior of neural networks. Moreover, in the also recent work by Favaro et al. 2025 , additional quantitative results were obtained for total variation and convex distances. However, these results were largely confined to the initialization regime. To this day, extensions to the full training trajectory remained limited, with few works addressing how approximation errors evolve over time or depend on architectural features such as width and depth. The present work builds on this gap by extending the quantitative convergence discussed above to trained networks, providing explicit bounds on the Wasserstein distance between the network output and the associated Gaussian process for any positive training time. From a spectral perspective, the NTK’s conditioning plays a central role in understanding convergence rates and generalization. Lower bounds on the smallest eigenvalue of the empirical NTK have been derived under various conditions. For instance, Karhadkar et al. 2024 and Bombari et al. 2022 provide sharp bounds in the context of ReLU and smooth activation functions, respectively. Additionaly, Carvalho et al. 2025 showed that under very mild assumptions on the non-linearity and non-proportionality of the training data, the analytic NTK is not degenerate. These results are essential for establishing the stability of the gradient flow and, hence, for deriving quantitative convergence guarantees. Our results are closely related to the work of de G. Matthews et al. 2018 , who proved weak convergence of fully-connected BNNs at initialization to a Gaussian process under the metric \rho(f,f^{\prime})=\sum_{i\in\mathbb{N}}2^{-i}\min\{1,|f(x_{i})-f^{\prime}(x_{i})|\} , defined on a countable input set. In our setting, the input set is finite; considering the restriction \rho_{F} , it follows that convergence in \mathcal{W}_{2} implies convergence

Read the complete text