Convergence of Stochastic Gradient Langevin Dynamics in the Lazy Training Regime

Noah Oberweis [email protected], Semih Cayci [email protected] · 2025

Read Convergence of Stochastic Gradient Langevin Dynamics in the Lazy Training Regime by Noah Oberweis [email protected], Semih Cayci [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21245.txt.

Introduction

Noah Oberweis and Semih Cayci analyze a training method that mixes gradient descent with random noise, viewed through a continuous-time equation. They focus on the lazy training regime, where a wide network barely moves from its starting values while it learns, a setting that makes the math tractable.

Under stated smoothness conditions on the loss, they show the process keeps a well-behaved kernel with high probability throughout training. They also prove fast, exponential convergence toward the best fit to the training data, in expectation, with explicit bounds tied to time and network width. Numerical examples in a regression setting match the theory, supporting the claim that noisy training stays stable in this regime.

Catalog record

CREATORNoah Oberweis [email protected], Semih Cayci [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21245

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Convergence of Stochastic Gradient Langevin Dynamics in the Lazy Training Regime Noah Oberweis [email protected], Semih Cayci [email protected] Abstract Continuous-time models provide important insights into the training dynamics of optimization algorithms in deep learning. In this work, we establish a non-asymptotic convergence analysis of stochastic gradient Langevin dynamics (SGLD), which is an Itô stochastic differential equation (SDE) approximation of stochastic gradient descent in continuous time, in the lazy training regime. We show that, under regularity conditions on the Hessian of the loss function, SGLD with multiplicative and state-dependent noise (i) yields a non-degenerate kernel throughout the training process with high probability, and (ii) achieves exponential convergence to the empirical risk minimizer in expectation, and we establish finite-time and finite-width bounds on the optimality gap. We corroborate our theoretical findings with numerical examples in the regression setting. 1 Introduction Stochastic gradient descent (SGD) has been the main workhorse of deep learning due to its low computational complexity and effectiveness. The existing theoretical analyses of first-order optimization methods in deep learning predominantly study the deterministic (full-batch) gradient descent, which neglect the impact of inherent stochasticity of SGD in training neural networks. Although there exist recent works that investigate stochastic dynamics (see, for example, Lugosi & Nualart 2024 ), explicit finite-time convergence rates for the stochastic first-order methods under realistic and interpretable conditions still remain elusive, which motivates our work. In this work, we establish theoretical guarantees for stochastic gradient Langevin dynamics (SGLD), a continuous-time approximation of SGD, in the lazy training regime. One important application of our theoretical analysis is finite-time and finite-width bounds on the training loss for deep feedforward neural networks trained by SGLD in a supervised learning setting. The main contributions of our work include the following: • Exponential convergence rates in expectation: By leveraging tools from stochastic calculus, we derive a stochastic Grönwall-type inequality for SDEs in the lazy training regime, which establishes an exponential decay rate for the expected optimality gap as long as the parameters are within a certain neighborhood of the initialization. To that end, we explicitly characterize the impact of the curvature of the loss function in the parameter space and the output scaling factor on the effective noise during the training process. The resulting scaling law describes sufficient conditions on the curvature of a global minimum to prevent SGLD from escaping it, which is one of the key outcomes of our analysis. • High probability bounds on the conservation of the lazy training regime: A key quantity in the lazy training analysis is the first-exit time at which the parameters leave a certain neighborhood of their random initialization, which lower bounds the duration of descent throughout the optimization steps. Leveraging our stochastic error analysis, we derive high-probability bounds on this first-exit time, and show that, for sufficiently large output scaling, the parameters remain in this neighborhood indefinitely with high probability, thereby completing the convergence argument in the stochastic setting. • Applications in deep learning: We apply our theoretical analysis to derive explicit non-asymptotic (finite-time and finite-width) bounds on training loss for deep neural networks trained by SGLD in the lazy training regime. To the best of our knowledge, this is the first analysis of stochastic gradient Langevin dynamics in this regime. Our work builds on and extends the deterministic lazy training analysis in Chizat et al. 2020 , which analyzes the convergence of gradient flow for overparameterized neural networks in the lazy training (or the so-called kernel) regime. In Chizat et al. 2020 , it was demonstrated that the lazy training phenomenon arises from an appropriately chosen output scaling factor, which ensures that the neural network parameters remain in a small enough neighborhood of their random initialization. We note that Chizat et al. 2020 considers deterministic (full-batch) gradient flow in the lazy training regime, and does not address the stochasticity that stems from stochastic subsampling in SGD, which is crucial for learning with large datasets. In this work, we extend this framework by analyzing a stochastic continuous-time approximation of the SGD. 1.1 Related Work Convergence of overparameterized neural networks in the kernel regime. Overparameterized neural networks have been shown to achieve interpolation, which implies zero training error when they are trained with first-order methods despite the highly non-convex optimization landscape. At the same time, they achieve impressive generalization performance even with noisy data ( Belkin et al. 2019 ; Bartlett et al. 2020 ; Zhang et al. 2021 ) . Additionally, works such as Oymak & Soltanolkotabi 2019 have demonstrated that the overparameterized regime can be reached with fewer parameters than previously suggested by the theory. The theoretical explanation of this phenomenon has been a focal point of interest. It was shown in Jacot et al. 2020 ; Du et al. 2019b ; Li & Liang 2018 that randomly-initialized overparameterized neural networks, trained with gradient descent, achieve global optimality without moving away from their random initialization, which is known as the lazy training or kernel regime. The main inspiration for this work is Chizat et al. 2020 , where lazy training is first analyzed as a consequence of an artificial scaling factor. The existing works in the lazy training regime predominantly analyze overparameterized neural networks trained with (full-batch) gradient descent, and they do not extend to stochastic gradient descent, which is the core focus of our work. Additionally, it is essential to point out that, contrary to other works, our primary goal is not the analysis of linearized dynamics defined by \bar{h}(\omega):=h(\omega_{0})+Dh(\omega_{0})(\omega-\omega_{0}) . It can be shown that gradient flow stays arbitrarily close to the linearized dynamics for sufficiently large \alpha . We will use very similar techniques to prove an exponential convergence rate of the empirical error for a sufficiently large \alpha without looking into the case where \alpha\rightarrow\infty . Convergence of stochastic gradient descent for neural networks. Dynamics of stochastic gradient descent have been investigated in Lugosi & Nualart 2024 for deep linear neural networks, which shows that a global optimum is achieved under some regularity assumptions on the loss landscape with a convergence rate that has an implicit dependency on the problem parameters. In our work, inspired by the analysis in Lugosi & Nualart 2024 , we analyze the stochastic gradient dynamics for training non-linear neural networks, and establish exponential convergence to the global optimum with an explicit and interpretable convergence rate under easily verifiable regularity assumptions. Modeling of discrete training algorithms by SDEs: Continuous-time modeling of discrete-time stochastic gradient descent by using SDEs has recently been investigated in many works, as the SDE model provides essential insights into the discrete stochastic gradient descent training algorithm. In this work, we consider an SDE-model to approximate stochastic gradient descent dynamics, inspired by Li et al. 2017 , which proved the viability of using stochastic gradient Langevin dynamics to model mini-batch stochastic gradient descent. Under certain assumptions, Li et al. 2017 shows that mini-batch stochastic gradient descent is a weak order-one approximation of the stochastic gradient Langevin dynamics equation 1 (see Remark 1 ). We limit our results to continuous models (as defined in the following section), although training typically employs stochastic gradient descent or other iterative methods in most practical applications. In addition to the multiple technical advantages arising from the continuous

Read the complete text