Understanding the role of depth in the neural tangent kernel for overparameterized neural networks
William St-Arnaud, Margarida Carvalho, Golnoosh Farnadi · 2025
Read Understanding the role of depth in the neural tangent kernel for overparameterized neural networks by William St-Arnaud, Margarida Carvalho, Golnoosh Farnadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07272.txt.
Introduction
William St-Arnaud, Margarida Carvalho, and Golnoosh Farnadi revisit the neural tangent kernel. The theory says infinitely wide networks trained by gradient descent behave like kernel machines stuck in a lazy regime where nothing internal changes. Two accepted conclusions say the theory fails at large width, because hidden activations supposedly freeze, and at large depth, because the kernel's prediction becomes data-independent and trivial. The authors challenge both.
Experimentally, while each single neuron's activation change vanishes, the aggregate norm of those changes does not, and a theorem confirms the last hidden layer's activations truly move. They further show that scaling depth and stopping time properly in wide ReLU networks yields well-behaved, non-trivial outputs on large data sets. The results reopen the question of what the kernel picture can explain about real networks.
Catalog record
| CREATOR | William St-Arnaud, Margarida Carvalho, Golnoosh Farnadi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07272
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding the role of depth in the neural tangent kernel for overparameterized neural networks William St-Arnaud, Margarida Carvalho, Golnoosh Farnadi Abstract Overparameterized fully-connected neural networks have been shown to behave like kernel models when trained with gradient descent, under mild conditions on the width, the learning rate, and the parameter initialization. In the limit of infinitely large widths and small learning rate, the kernel that is obtained allows to represent the output of the learned model with a closed-form solution. This closed-form solution hinges on the invertibility of the limiting kernel, a property that often holds on real-world datasets. In this work, we analyze the sensitivity of large ReLU networks to increasing depths by characterizing the corresponding limiting kernel. Our theoretical results demonstrate that the normalized limiting kernel approaches the matrix of ones. In contrast, they show the corresponding closed-form solution approaches a fixed limit on the sphere. We empirically evaluate the order of magnitude in network depth required to observe this convergent behavior, and we describe the essential properties that enable the generalization of our results to other kernels. 1 Introduction Machine learning approaches have demonstrated a remarkable ability in helping solve a multitude of problems across a large span of tasks. Whether the task is classification, prediction, or image generation, some variants of the combination of a neural network plus gradient descent have managed to achieve superhuman ability in some instances ( 11 ; 20 ; 22 ) . In recent years, a particular observation has been made regarding neural networks that are overparameterized. While previous common thought indicated that these large networks would fall prey to overfitting, this conclusion is being challenged empirically ( 4 ) . This phenomenon is not quite well understood and it has led to works analyzing the learning dynamics of overparameterized models updated with gradient descent ( 16 ; 15 ; 10 ) . While these works provide insight as to the learning dynamics of fully-connected neural networks that are overparameterized, 10 , in particular, offer a closed-form solution to the gradient flow based on a kernel that is recursively computed. This result offers the possibility of approximately predicting the output of an overparameterized neural network learned through gradient descent, without explicitly training the model. This comes at the cost of computing a kernel over a particular dataset, which involves the computation of expectations. We focus on fully-connected ReLU networks, which allows us to both speed up the computation of the kernel and obtain a more interpretable closed-form solution. Leveraging this more interpretable closed-form solution, we study the role of depth in the limiting kernel of infinitely wide fully-connected ReLU networks. Our contributions address two central aspects of the effect of increasing depths: 1) the convergence of the kernel, established in Proposition 4 , and 2) the limiting solution to the output of a fully-connected ReLU network under infinitely wide hidden layers and infinitely small learning rate via Theorem 3 . Our results apply to arbitrary data with support on the sphere and, in contrast to previous literature, do not require any assumptions on the spectrum of the Hermite expansion or the Mercer decomposition of the kernel. In contrast to 9 , we study the deterministic limit of the neural tangent kernel when the width is much larger than the depth: while we allow the depth to increase to infinity, the rate at which it does so is much slower than the widths of the hidden layers of a neural network. We offer a summary of results in relation to ours in table 1 in Appendix D . 2 Related Work The learning dynamics of overparameterized neural networks have been extensively studied through the lenses of kernel methods and Hessian-based analysis. A key development in this area is the neural tangent kernel (NTK), introduced by 10 , which describes how infinitely wide fully-connected neural networks trained with gradient descent evolve linearly in function space. The NTK framework formalizes how, under common assumptions—particularly Gaussian initialization and wide-layer limits—neural networks behave similarly to kernel methods during training. 3 extend the NTK to convolutional neural networks (CNN), further highlighting the framework’s capacity to capture learning dynamics. Subsequent work has reinforced the kernel-based interpretation of training dynamics through analyses of the Hessian matrix. 16 ; 15 and 4 demonstrate that the loss landscape of overparameterized neural networks often exhibits near-linearity, with low-curvature regions and small-norm Hessians, supporting the NTK-based approximation. These findings suggest that network outputs are relatively stable during training, especially for wide architectures with standard initialization. 12 provide an in-depth empirical analysis of NTK models and their performance compared to finite-width neural networks of various architectures (e.g. fully-connected, CNNs). One key observation is that NTKs often outperform finite-width networks, yet are usually surpassed by conventional CNNs. The NTK typically requires Monte Carlo estimation of expectations over Gaussian distributions, especially when nonlinearities from activations are involved. This reliance on sampling can be computationally expensive and introduces variance in the resulting kernel evaluations. However, closed-form expressions have been derived for certain activation functions, such as ReLU and leaky ReLU ( 21 ) . Additionally, while prior work has largely focused on width, less attention has been paid to how depth affects NTK sensitivity to initialization and the associated multiplicity in network outputs. Among the few works addressing this dimension, 5 show that for the uniform measure on the sphere, the reproducing kernel (c.f. reproducing kernel Hilbert space or RKHS) leads to the same representation power regardless of the network depth. This raises the question of the significance of depth for the NTK. Further insights are provided by 19 , who derive an asymptotic lower bound on the smallest eigenvalue of the NTK through the Hermite expansion of the kernel. This result can be used to derive bounds on the generalization of the model ( 2 ) and gives a better grasp on understanding the role that depth plays in both convergence and generalization. 18 characterize the full spectrum of the NTK via the Hermite expansion for arbitrary datasets on the sphere. They recover an empirical observation that the eigenvalues of the NTK follow power law decay with respect to the size of the training set; 14 extend this line of results to general domains beyond the sphere. In addition to the previous result from 10 regarding the convergence to kernel regression in the infinite width case, they also establish a uniform convergence bound to the NTK regressor for the output of the trained model. This improves upon the pointwise convergence bounds found in 13 ; 3 ; 1 . As in the initial paper by 10 , the limiting kernel is observed to be deterministic. This is contrasted with the work of 9 , where the authors characterize the NTK directly through the ratio of its second and first moment. They obtain a limiting result showing that if the ratio of depth to width is unbounded, the NTK has a much higher variance than mean, implying that it is highly stochastic. 3 Notation To highlight the role of the depth L of a neural network, we denote elements that depend on layer l\in\{1,\ldots,L\} with a superscript (l) , e.g. \kappa^{(l)} . To emphasize the special role