On the rate of convergence of an over-parametrized deep
neural network regression estimate learned by gradient · 2025
Read On the rate of convergence of an over-parametrized deep by neural network regression estimate learned by gradient (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03405.txt.
Introduction
Michael Kohler proves a 2025 result on how fast a deep neural network fit nears the true curve when all weights learn by gradient descent. The setup is nonparametric regression with random input points. Error is scored by the squared gap averaged over the input spread.
The network is wider than the data demand, a case called over-parametrized, and uses a logistic switch in its units. For true curves with a set grade of smoothness, the fit reaches a near-best rate of approach. The proof ties the learning path of descent to that rate in full.
Catalog record
| CREATOR | neural network regression estimate learned by gradient |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03405
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the rate of convergence of an over-parametrized deep neural network regression estimate learned by gradient Abstract April 4, 2025 Abstract Nonparametric regression with random design is considered. The L_{2} error with integration with respect to the design measure is used as the error criterion. An over-parametrized deep neural network regression estimate with logistic activation function is defined, where all weights are learned by gradient descent. It is shown that the estimate achieves a nearly optimal rate of convergence in case that the regression function is (p,C) –smooth. AMS classification: Primary 62G08; secondary 62G20. Key words and phrases: Deep neural networks, gradient descent, over-parametrization, rate of convergence, regression estimation. 1 Introduction 1.1 Scope of this paper Deep learning has achieved tremendous success in applications, e.g., in image classification (cf., e.g., Krizhevsky, Sutskever and Hinton (2012)), language recognition (cf., e.g., Kim (2014)) machine translation (cf., e.g., Wu et al. (2016)), mastering of games (cf., e.g., Silver et al. (2017)) or simulation of human conversation (cf., e.g., Zong and Krishnamachari (2022)). From a theoretical point of view this great success is still a mystery. In particular, it is unclear why the use of over-parametrized deep neural networks, which have much more weights than there are data points, does not lead to an overfitting of the estimate, and why gradient descent is able to minimize the nonlinear and non-convex empirical risk in the definition of the estimates in such a way that the estimates can achieve a small risk for randomly initialized starting values for the weights of the networks. In a standard regression setting we give answers to these two questions in the special situation that we want to estimate a (p,C) –smooth regression function. 1.2 Nonparametric regression We study deep neural networks in the context of nonparametric regression. Here we have given an \mathbb{R}^{d}\times\mathbb{R} –valued random vector (X,Y) with {\mathbf{E}}Y^{2}<\infty , and our goal is to predict the value of Y given the value of X . Let m(x)={\mathbf{E}}\{Y|X=x\} be the so–called regression function. Then any measurable f:\mathbb{R}^{d}\rightarrow\mathbb{R} satisfies {\mathbf{E}}\{|f(X)-Y|^{2}\}={\mathbf{E}}\{|m(X)-Y|^{2}\}+\int|f(x)-m(x)|^{2}{\mathbf{P}}_{X}(dx) (1) (cf., e.g., Section 1.1 in Györfi et al. (2002)), hence in view of minimizing the so-called L_{2} risk ( 1 ) of f the regression function m is the optimal predictor, and the so–called L_{2} error \int|f(x)-m(x)|^{2}{\mathbf{P}}_{X}(dx) (2) describes how far the L_{2} risk of a function f is away from its optimal value. In applications typically the distribution of (X,Y) and hence also the corresponding regression function m is unknown. But often it is possible to observe data from the underlying distribution, and the task is to use this data to estimate the unknown regression function. In view of minimization of the L_{2} risk of the estimate, here it is natural to use the L_{2} error as an error criterion. In order to introduce this problem formally, let (X,Y) , (X_{1},Y_{1}) , …, (X_{n},Y_{n}) be independent and identically distributed. In nonparametric regression the data set {\mathcal{D}}_{n}=\left\{(X_{1},Y_{1}),\dots,(X_{n},Y_{n})\right\} (3) is given, and the task is to construct an estimate m_{n}(\cdot)=m_{n}(\cdot,{\mathcal{D}}_{n}):\mathbb{R}^{d}\rightarrow\mathbb{R} such that its L_{2} error \int|m_{n}(x)-m(x)|^{2}{\mathbf{P}}_{X}(dx) is small. A systematic introduction to nonparametric regression, its estimates and known results can be found, e.g., in Györfi et al. (2002). 1.3 Rate of convergence Stone (1982) determined the optimal Minimax rate of convergence of the L_{2} error in case of a smooth regression function. Here he considered so-called (p,C) –smooth regression functions, which are defined as follows. Definition 1 Let p=q+s for some q\in\mathbb{N}_{0} and 0<s\leq 1 . A function m:\mathbb{R}^{d}\rightarrow\mathbb{R} is called (p,C) -smooth, if for every \bm{\alpha}=(\alpha_{1},\dots,\alpha_{d})\in\mathbb{N}_{0}^{d} with \sum_{j=1}^{d}\alpha_{j}=q the partial derivative \partial^{q}m/(\partial x_{1}^{\alpha_{1}}\dots\partial x_{d}^{\alpha_{d}}) exists and satisfies \left|\frac{\partial^{q}m}{\partial x_{1}^{\alpha_{1}}\dots\partial x_{d}^{\alpha_{d}}}(x)-\frac{\partial^{q}m}{\partial x_{1}^{\alpha_{1}}\dots\partial x_{d}^{\alpha_{d}}}(z)\right|\leq C\cdot\|\mathbb{x}-\mathbb{z}\|^{s} for all \mathbb{x},\mathbb{z}\in\mathbb{R}^{d} , where \|\cdot\| denotes the Euclidean norm. Stone (1982) showed that in case of a (p,C) –smooth regression function the optimal Minimax rate of convergence for the expected L_{2} error is n^{-\frac{2p}{2p+d}}. This rate suffers from the so–called curse of dimensionality: If the dimension d is large compared to the smoothness p of the regression function, the exponent will be close to zero and the rate of convergence will be rather slow. Since this rate is optimal, the only way to circumvent this is to impose additional assumptions on the structure of the regression function. For this, various assumptions exists, e.g., additive models (cf., e.g., Stone (1985)), interaction models (cf., e.g., Stone (1994)), single index models (cf., e.g., Härdle, Hall and Ichimura (1993), Härdle and Stoker (1989), Yu and Ruppert (2002) and Kong and Xia (2007)) or projection pursuit (cf, e.g., Friedman and Stuetzle (1981)), where corresponding low dimensional rates of convergence can be achieved (cf., e.g., Stone (1985, 1994) and Chapter 22 in Györfi et al. (2002)). 1.4 Least squares neural network estimates One way to estimate a regression function is to define a function space {\cal F}_{n} consisting of functions f:\mathbb{R}^{d}\rightarrow\mathbb{R} and to use the principle of least squares to select one of its functions as the regression estimate, i.e., to define m_{n}(\cdot)=\arg\min_{f\in{\cal F}_{n}}\frac{1}{n}\sum_{i=1}^{n}|f(X_{i})-Y_{i}|^{2}. In view of the L_{2} error of the estimate it is important that the function space is on the one hand large enough such that a function is contained in it which approximates the unknown regression function well, and that on the other hand the function space is not too complex so that the empirical L_{2} risk \frac{1}{n}\sum_{i=1}^{n}|f(X_{i})-Y_{i}|^{2} is not too far from the L_{2} risk for functions in this function space. One possible way to define such function spaces in case of d large is to use feedforward neural networks. These function depend on an activation function \sigma:\mathbb{R}\rightarrow\mathbb{R} , e.g., \sigma(x)=\max\{x,0\} (so-called ReLU-activation function) or \sigma(x)=\frac{1}{1+e^{-x}} (so-called logistic activation function). The most simple form of neural networks are shallow networks, i.e., neural networks with one hidden layer, in which a simple linear combination of artifical neurons defined by applying the activation function to a linear combination of the components of the input is used to define a function f:\mathbb{R}^{d}\rightarrow\mathbb{R} by f(x)=\sum_{k=1}^{K}\alpha_{k}\cdot\sigma\left(\sum_{j=1}^{d}\beta_{k,j}\cdot x^{(j)}+\beta_{k,0}\right)+\alpha_{0}. (4) Here K\in\mathbb{N} is the number of neurons, and the weights \alpha_{k}\in\mathbb{R} (k=0,\dots,K) , \beta_{k,j}\in\mathbb{R} (k=1,\dots,K,j=0,\dots,d) are chosen by the principle of least squares. The rate of convergence of shallow neural networks regression estimates has been analyzed in Barron (1994) and McCaffrey and Gallant (1994). Barron (1994) proved a dimensionless rate of n^{-1/2} (up to some logarithmic factor), provided the Fourier transform of the regression function has a finite first moment, which basically requires that the function becomes smoother with increasing dimension d of X . McCaffrey and Gallant (1994) showed a rate of n^{-\frac{2p}{2p+d+5}+\varepsilon} in case of a (p,C) -smooth regression function, but their study was restricted to the use of a certain cosine squasher as activation function. In deep learning neural networks with several hidden layers are used to define classes of functions. Here a feedforward neural network with L\in\mathbb{N} hidden layers and k_{s}\in\mathbb{N} neurons in the layers s\in\{1,\dots,L\} is recursively defined by f_{\mathbf{w}}(x)=\sum_{j\in\{1,\dots,k_{L}\}}w_{1,j}^{(L)}\cdot f_{j}^{(L)}(x), (5) where f_{i}^{(s)}(x)=\sigma\left(\sum_{j\in\{1,\dots,k_{s-1}\}}w_{i,j}^{(s-1)}\cdot f_{j}^{(s-1)}(x)+w_{i,0}^{(s-1)}\right)\quad\mbox{for }s\in\{2,\dots,L\}\mbox{ and }i>0 (6) and f_{i}^{(1)}(x)=\sigma\left(\sum_{j\in\{1,\dots,d\}}w_{i,j}^{(0)}\cdot x^{(j)}+w_{i,0}^{(0)}\right)\quad\mbox{for }i>0. (7) The rate of convergence of least squares estimates based on multilayer neural networks has been analyzed in Kohler and Krzyżak (2017), Imaizumi and Fukamizu (2018), Bauer and Kohler (2019), Suzuki and Nitanda (2019), Schmidt-Hieber (2020) and Kohler and Langer (2021). One of the main results