Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime

Raphaël Barboni, Gabriel Peyré, François-Xavier Vialard · 2025

Read Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime by Raphaël Barboni, Gabriel Peyré, François-Xavier Vialard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18208.txt.

Introduction

Raphael Barboni, Gabriel Peyre, and Francois-Xavier Vialard studied in 2025 how two-layer neural networks learn features at a very fast pace. Training such networks is a hard, non-convex problem in high dimensions. Many past results either give no rates or freeze the learned features in place.

Their method removes the linear part of the problem and trains only the nonlinear features, a trick long used for split least-squares problems. In a teacher-student setup, they prove rates for how the learned feature spread approaches the teacher spread. As a smoothing term fades, the motion matches a very fast diffusion equation. Known results on such equations then give hard numbers for the approach.

Catalog record

CREATORRaphaël Barboni, Gabriel Peyré, François-Xavier Vialard
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE153 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18208

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime Raphaël Barboni, Gabriel Peyré, François-Xavier Vialard Abstract We study the convergence of gradient methods for the training of mean-field single-hidden-layer neural networks with square loss. For this high-dimensional and non-convex optimization problem, most known convergence results are either qualitative or rely on a neural tangent kernel analysis where nonlinear representations of the data are fixed. Using that this problem belongs to the class of separable nonlinear least squares problems, we consider here a Variable Projection (VarPro) or two-timescale learning algorithm, thereby eliminating the linear variables and reducing the learning problem to the training of nonlinear features. In a teacher-student scenario, we show such a strategy enables provable convergence rates for the sampling of a teacher feature distribution. Precisely, in the limit where the regularization strength vanishes, we show that the dynamic of the feature distribution corresponds to a weighted ultra-fast diffusion equation . Recent results on the asymptotic behavior of such PDEs then give quantitative guarantees for the convergence of the learned feature distribution. 1 Introduction Machine learning methods based on artificial neural networks have recently experienced a significant increase in popularity due to their efficiency in solving numerous supervised or unsupervised learning tasks. This success owes to their capacity to perform feature learning , that is to extract meaningful representations from the data during the training process [ 35 , Chap. 15] , standing in contrast with kernel methods for which feature representations are designed by hand and fixed during training [ 44 ] . Feature learning is believed to play a fundamental role in the generalization performance of neural networks. For example, adaptivity to low-dimensional representations of the data can prevent the curse of dimensionality [ 5 , 31 ] . However, the process through which features are learned remains largely misunderstood. Indeed, adaptivity of the representations comes in neural networks at the price of a nonlinear parameterization, making the training dynamic more difficult to analyze. Specifically, for overparameterized neural network architectures where the dimension of the parameter space greatly exceeds the number of training samples, recent works have put forward the crucial role played by the choice of scaling w.r.t. the number of parameters in the training dynamic [ 22 , 56 , 91 ] . For single-hidden-layer neural networks, the “kernel regime”, corresponding to a scaling of 1/\sqrt{M} where M is the width, has been identified as a scaling for which the model is well-approximated by its linearization around initialization, therefore reducing to a kernel method [ 49 ] . Relying on the good conditioning of the “Neural Tangent Kernel (NTK)”, this regime provides convergence of gradient descent towards a global minimizer of the risk at a linear rate [ 1 , 27 , 54 , 92 ] . However, this regime has also been shown to suffer from a “lazy training” behavior preventing significant modification of the feature distribution and associated to poor generalization guarantees [ 22 ] . In contrast, another line of work has been focused on the “mean field” regime corresponding to a scaling of 1/M (which we consider in Eq. 1 ) for which the neural network is parameterized by a probability distribution over the space of weights [ 23 , 58 , 72 , 78 ] . While such a choice of scaling has been shown to enable nonlinear feature learning behaviors [ 91 ] , existing convergence results are primarily qualitative, lacking explicit convergence rates. To bridge this gap, we are interested in this work in the dynamic of the feature distribution in the training of mean-field models of shallow neural network architectures. We study more particularly a variable projection or two-timescale learning strategy which allows reducing the learning problem to the training of the feature distribution. 1.1 Mean-field neural networks and two-timescale learning We consider in this work neural networks with parameter in the parameter space \Omega , which is assumed to be either the n -dimensional torus \mathbb{T}^{n}=\mathbb{R}^{n}/\mathbb{Z}^{n} , or a closed, bounded and convex domain of \mathbb{R}^{n} . In the following, we are given a map \phi:\Omega\times\mathbb{R}^{d}\to\mathbb{R} , called feature map , and for an integer M\geq 1 , we define a single-hidden-layer (SHL) neural network of width M with inner weights \{\omega_{i}\}_{1\leq i\leq M}\in\Omega^{M} and outer weights \{u_{i}\}_{1\leq i\leq M}\in\mathbb{R}^{M} as the map: \displaystyle F_{\{(\omega_{i},u_{i})\}}:x\in\mathbb{R}^{d}\mapsto\frac{1}{M}\sum_{i=1}^{M}u_{i}\phi(\omega_{i},x)\in\mathbb{R}, (1) taking inputs in the input space \mathbb{R}^{d} and returning values in the output space \mathbb{R} . Thanks to the interchangeability of the indices and the normalisation factor 1/M , the above model can be reparameterized in terms of the empirical distribution of the inner weights \{\omega_{i}\}_{1\leq i\leq M} . Given an arbitrary probability distribution \mu\in\mathcal{P}(\Omega) on the space of inner weights and a measurable map u\in L^{1}(\mu) we define: \displaystyle F_{\mu,u}:x\in\mathbb{R}^{d}\mapsto\int_{\Omega}u(\omega)\phi(\omega,x)\mathrm{d}\mu(\omega)\in\mathbb{R}. (2) In particular, for the empirical distribution \hat{\mu}=\frac{1}{M}\sum_{i=1}^{M}\delta_{\omega_{i}} and the outer weights \hat{u}(\omega_{i})=u_{i} we recover the finite width SHL F_{\hat{\mu},\hat{u}}=F_{\{(\omega_{i},u_{i})\}} . Such a “mean-field” model of neural network has been proposed by several authors to study the training of neural networks at arbitrary large width [ 23 , 58 , 72 , 78 ] . Supervised learning In the context of supervised learning, training a neural network consists in minimizing a training risk associated to the evaluation of the model on some training data . We consider in this work a univariate regression setting where the neural network weights are trained for minimizing the mean square error with a target signal Y\in L^{2}(\rho) evaluated on training data with distribution \rho\in\mathcal{P}(\mathbb{R}^{d}) . For a regularization strength \lambda>0 and \mu\in\mathcal{P}(\Omega) , u\in L^{1}(\mu) we define the training risk as: \displaystyle\mathcal{R}^{\lambda}(\mu,u)\coloneqq\frac{1}{2}\|F_{\mu,u}-Y\|^{2}_{L^{2}(\rho)}+\lambda\|u\|^{2}_{L^{2}(\mu)}, (3) where we assume \mathcal{R}^{\lambda}(\mu,u)=+\infty if u\notin L^{2}(\mu) . Training the neural network then amounts to finding parameters (\mu,u)\in\argmin\mathcal{R}^{\lambda} . Example of applications Note that the mean-field neural network model of Eq. 2 can be seen as a linear model acting on (signed) measures. Indeed, for \mu\in\mathcal{P}(\Omega) and u\in L^{1}(\mu) , we have F_{\mu,u}=\Phi\star(u\mu) where for every finite Borel measure \nu\in\mathcal{M}(\Omega) we define: \displaystyle\Phi\star\nu\coloneqq\int_{\Omega}\phi(\omega,.)\mathrm{d}\nu(\omega)\,. (4) In turn, minimization of functionals of the form in Eq. 3 with linear models acting on the space of measures have numerous applications depending on the choice of the feature map \phi . • Two-layer perceptron: The perceptron model is arguably the prototypical example of a neural network. It consists in considering \Omega\subset\mathbb{R}^{d+1} and a feature map \phi:(\omega,x)\mapsto\sigma(\omega^{\top}\bar{x}) where \bar{x}=(x,1)\in\mathbb{R}^{d+1} and \sigma:\mathbb{R}\to\mathbb{R} is some nonlinear activation such as the Rectified Linear Unit (ReLU) or hyperbolic tangent. Owing to their great expressivity [ 24 ] , this class of models is ubiquitous in applications where an unknown signal is to be recovered from data observations. • Radial Basis Function (RBF) neural networks and signal deconvolution: RBF neural networks [ 68 , 51 ] is an example of a simple architecture in which the feature map consists of a translation invariant kernel k i.e. \Omega\subset\mathbb{R}^{d} and \phi:(\omega,x)\mapsto k(\omega-x) . The network F_{\mu,u} then implements a convolution with the kernel k and minimization of the risk \mathcal{R}^{\lambda} amounts to solve a form of deconvolution problem. This has important applications in signal processing where one wants to recover an unknown signal given noisy or filtered observations [ 25 , 28 ] . Training with gradient

Read the complete text