On the global convergence of gradient descent for wide shallow models with bounded nonlinearities
Romain Petit, Clarice Poon, Gabriel Peyré · 2026
Read On the global convergence of gradient descent for wide shallow models with bounded nonlinearities by Romain Petit, Clarice Poon, Gabriel Peyré (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10775.txt.
Introduction
Romain Petit, Clarice Poon, and Gabriel Peyré study why gradient descent, the workhorse trainer of neural networks, so often lands on the best possible fit despite a bumpy loss landscape. They focus on wide, shallow networks with bounded activation functions, a class that covers multi-head attention layers and two-layer sigmoid nets.
Building on earlier work, they prove that every non-global minimum is unstable under gradient descent. Training can therefore only settle on global minima when starting weights spread widely and the network is wide enough. They also finish a missing proof step about escaping sets and show the mean-field training dynamics stay well behaved. Theorists get a wider guarantee for a puzzling, widely observed success.
Catalog record
| CREATOR | Romain Petit, Clarice Poon, Gabriel Peyré |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 114883 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10775
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the global convergence of gradient descent for wide shallow models with bounded nonlinearities Romain Petit, Clarice Poon, Gabriel Peyré Abstract A surprising phenomenon in the training of neural networks is the ability of gradient descent to find global minimizers of the training loss despite its non-convexity. Following earlier works, we investigate this behavior for wide shallow networks. Existing results essentially cover the case of ReLU activations and the case of sigmoid activations with scalar output weights. We study a large class of models that includes multi-head attention layers and two-layer sigmoid networks with vector output weights. Building upon [ 10 ] , we prove that all non-global minimizers of the training loss are unstable under gradient descent dynamics. Thus, when the initial distribution of the parameters has full support (which includes the popular Gaussian case), and in the many hidden neurons or attention heads limit, continuous-time gradient descent can only converge to global minimizers. Establishing the instability of non-global minimizers corresponds to the construction of an “escaping active set” – we complete the proof of [ 10 ] to construct this set for models with bounded nonlinearities and scalar output weights. We also extend this construction to new cases for models with vector output weights. Finally, we show the well-posedness and the stability with respect to discretization of the mean field training dynamic for sub-Gaussian initializations. 1 Introduction The training of neural networks leads to the minimization of objective functions that are highly non-convex. As a result, first order optimization algorithms such as gradient descent could fail to reach an approximate minimizer and converge to non-optimal critical points. In certain settings, it has been observed that this behavior is not typical and that gradient descent finds global minimizers of the training loss despite its non-convexity. Among the various explanations that have been proposed, a line of works has focused on the study of the over-parameterized regime . The landmark results of [ 10 ] (see also [ 27 , 32 ] ) show that, if the model is sufficiently over-parameterized and the initialization densely covers the parameter space, then (continuous-time) gradient descent can only converge to global minimizers of the training loss. In these works, the over-parameterized models under study are typically fully connected neural networks with a single hidden layer and a large number of hidden neurons. Existing results essentially cover sigmoid activations with scalar output weights and ReLU activations. Besides networks with sigmoid activations and vector output weights, other popular architectures are not covered by these results. This is in particular the case of attention layers, which have become ubiquitous in recent years. In this work, we make a step towards bridging this gap by investigating global convergence results for a large class of over-parameterized models. 1.1 Main problem We study the training of parameterized models of the form (1/m)\sum_{i=1}^{m}\Phi(w_{i},\theta_{i}) , where \Phi is linear in the w variable. This setting covers two main examples of interest: transformers with a single multi-head attention layer and two-layer fully connected neural networks. In the first case, the w and \theta variables respectively correspond to the value matrix and to the query and key matrices. In the second case, they respectively correspond to the parameters of the output layer and to the parameters of the hidden layer. We analyze the training of such models with continuous-time gradient descent (or gradient flow). To be more precise, we analyze the gradient flow of the function F_{m}\colon\Omega^{m}\to\mathbb{R} defined by F_{m}((w_{i},\theta_{i})_{1\leqslant i\leqslant m})=\textstyle R((1/m)\sum_{i=1}^{m}\Phi(w_{i},\theta_{i})), (1) where \Omega=\mathbb{R}^{d_{w}}\times\mathbb{R}^{d_{\theta}} and R is a given loss function. The objective F_{m} is highly non-convex, and the iterates of gradient descent could get trapped by spurious local minima. Yet, in the large m (or over-parameterized ) regime, gradient descent often converges to global minimizers of F_{m} . The main objective of this work is to understand this phenomenon. Following [ 10 ] , the key object of our analysis is the dynamic one obtains by encoding the parameters of the model as a probability distribution over the parameter space \Omega . This evolution turns out to be a Wasserstein gradient flow (see [ 33 ] for an introduction to this topic) of a function F defined on the set \mathcal{P}(\Omega) of probability distributions on \Omega . By studying the long-time limit of this evolution, we aim to find conditions under which the gradient flow of F_{m} converges to a global minimizer. 1.2 Related work Scaling limits. It is by now well understood that non-convexity is not necessarily an obstacle when training highly over-parameterized neural networks, including deep ones. Under suitable scaling regimes, local and sometimes global convergence can be established. Different scalings lead to distinct limiting descriptions of training dynamics, with the two most studied cases being the Neural Tangent Kernel (NTK) regime and the mean-field or “feature learning” regime. These asymptotic viewpoints clarify why optimization succeeds in over-parameterized models, although they may not always align with practical behavior [ 11 , 25 , 36 ] . Wide networks and NTK. For nonlinear multilayer perceptrons, a standard scaling is to initialize weights with variance proportional to 1/\sqrt{m} , where m is the width of a hidden layer. In this setting, when the width grows, hidden units move little during training, and in the infinite-width limit, they remain fixed. The resulting dynamics correspond to kernel regression with the Neural Tangent Kernel [ 21 ] . This framework provides a route to proving global convergence in sufficiently wide networks, with refinements depending on data separability and network depth. For two-layer ReLU networks, convergence has been analyzed under the NTK scaling [ 24 ] . For deeper models, general convergence results have been established [ 1 , 14 , 25 , 37 , 23 , 9 , 29 ] . This analysis can be refined for residual networks, illustrating how skip connections stabilize training [ 19 , 5 , 38 , 4 ] . Feature learning regime and mean-field models. An alternative scaling is to initialize weights with variance 1/m . In this case, hidden units evolve significantly during training, leading to the so-called mean-field or feature learning regime. Here, the network can be described by a probability measure over neurons, which adapts during training. Unlike NTK, this regime captures representation learning and explains the ability of neural networks to exploit low-dimensional structures in data [ 3 , 18 ] . It also connects neural networks to functional analysis, where associated function spaces can be defined [ 15 ] . The mean-field scaling has been rigorously formalized as the limit of interacting particle systems, yielding a partial differential equation (PDE) of McKean–Vlasov type, which is a gradient flow for the Wasserstein metric [ 10 , 27 , 32 , 35 , 28 ] . This PDE viewpoint allows one to prove global convergence results when the support of the initial distribution is sufficiently broad [ 10 ] . The main goal of our work is to extend the PDE framework (without diffusion and noise injection) and convergence analysis to a broader class of architectures—so as to accommodate attention heads and vector output weights for bounded activation functions—as well as to more general initialization schemes, including non–compactly supported distributions such as Gaussians and,