Issues with Neural Tangent Kernel Approach to Neural Networks

Haoran Liu, Anthony Tai, David J. Crandall, Chunfeng Huang · 2025

Read Issues with Neural Tangent Kernel Approach to Neural Networks by Haoran Liu, Anthony Tai, David J. Crandall, Chunfeng Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10929.txt.

Introduction

Four researchers question a popular math shortcut for understanding neural networks, in a 2025 paper. The shortcut, the neural tangent kernel, treats a trained network as a form of kernel regression, a classic prediction method. A famed result claims the two are equivalent, which would make networks much easier to analyze. The team revisits the proof with care and tests the claim in numbers.

In their tests, adding a layer to a network and updating the kernel did not produce matching changes in prediction error. A plain kernel method that ignores training altogether scored about as well as the network-linked kernel. The authors conclude that the equivalence breaks down in practice. That gap, they argue, puts in doubt whether the kernel view truly captures how networks learn.

Catalog record

CREATORHaoran Liu, Anthony Tai, David J. Crandall, Chunfeng Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-10929

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Issues with Neural Tangent Kernel Approach to Neural Networks Haoran Liu, Anthony Tai, David J. Crandall, Chunfeng Huang Abstract Neural tangent kernels (NTKs) have been proposed to study the behavior of trained neural networks from the perspective of Gaussian processes. An important result in this body of work is the theorem of equivalence between a trained neural network and kernel regression with the corresponding NTK. This theorem allows for an interpretation of neural networks as special cases of kernel regression. However, does this theorem of equivalence hold in practice? In this paper, we revisit the derivation of the NTK rigorously and conduct numerical experiments to evaluate this equivalence theorem. We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error. Furthermore, we observe that kernel regression with a Gaussian process kernel in the literature that does not account for neural network training produces prediction errors very close to that of kernel regression with NTKs. These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks. 1 Introduction Recent advancements in generative models such as ChatGPT ( OpenAI, 2023 ) and MidJourney ( Midjourney, 2023 ) have sparked an uptick in discussions of AI, including areas of adoption and potential increases in efficiency. However, these advances in technology are not without risks. The Database of AI Litigation ( GWU, 2024 ) cites a number of AI-related lawsuits covering a wide variety of issues including copyright infringement in the training data, unfair competition via coordinated rent manipulation, misrepresentation of the safety of autonomous systems, and the lack of human review in automated decision making (health insurance, fraud detection, and job interviews). The fundamental building block of many advanced AI systems are neural networks. At a high level, neural networks are complex nonlinear models that learn from data. They are often used for prediction, dimension reduction, or generative tasks. While neural networks have shown to perform well at certain tasks, a limitation of such models is the lack of meaningful interpretation. This makes it difficult to understand precisely why these models make the decisions that they do and makes it difficult to come up with regulations or guidelines for safe, ethical, and trustworthy implementations. New technologies do require trial and error before becoming well-refined tools. Nevertheless, the importance of theoretical understanding of these models should not be dismissed. One popular framework for understanding the behavior of neural networks is the “Gaussian process view” ( Arora et al. (2019) , Jacot et al. (2018) , Lee et al. (2017) , Neal (1996) ). This framework addresses the behavior of neural networks as the widths of hidden layers become large. Neal (1996) showed that a pre-trained (parameters initialized but not trained) neural network with a single hidden layer converges to a Gaussian process as the width of the hidden layer approaches infinity. Lee et al. (2017) extends the work of Neal (1996) and showed that for a pre-trained fully connected neural network with arbitrary number of hidden layers ( \geq 1 ) converges to a Gaussian process as the widths of the hidden layers approach infinity. Jacot et al. (2018) developed the neural tangent kernel (NTK) to describe trained neural networks under some assumptions. The key claim is that changes to the network parameters can be described by a random kernel which converges, as training time goes to infinity, to a deterministic kernel: the neural tangent kernel ( Jacot et al., 2018 ) . Arora et al. (2019) extends the neural tangent kernels to include convolutional operations and proposed a theorem of equivalence between trained neural networks and kernel regression predictor using the corresponding NTKs. This theorem of equivalence allows one to interpret neural networks as special cases of kernel regression. The body of work on neural tangent kernels have been extended to recurrent neural networks ( Alemohammad, 2022 ) and attention-based models ( Hron et al., 2020 ) as well. 1.1 Related Works There are some existing critiques of the NTK. Simon et al. (2022) argued that increasing the layers in the NTK formula does not provide additional kernel expressivity. This runs counter to the recursive nature of the NTK formula from Jacot et al. (2018) and Arora et al. (2019) where the number of recursive steps depends on the number of layers for the corresponding neural network. Chizat et al. (2019) showed through numerical experiments that the performance of convolutional neural networks (CNNs) used in practice degrade as they become “lazy” and hence the success of CNNs cannot be adequately explained by “lazy training.” Seleznova, Kutyniok (2022) showed that whether the lazy training assumption holds depends on the variance of parameter initialization and in the “chaotic phase” (situations when lazy training doesn’t hold), the NTK matrix does not stay constant during training. 1.2 Contributions We evaluate the theorem of equivalence ( Arora et al., 2019 ) between neural networks and kernel regression with the corresponding neural tangent kernel through numerical experiments. Our results show that kernel regression with the neural tangent kernel does not adequately address the behavior of trained neural networks. Furthermore, we show that kernel regression with the Gaussian process kernels (which correspond to initialized but untrained neural networks) has predictive performance very similar to that of kernel regression using the neural tangent kernels. 2 Background knowledge and methods used 2.1 Fully Connected Neural Networks In general, the i^{th} component of the output of a fully connected neural networks with \ell hidden layers can be expressed as: \displaystyle z_{i}^{(\ell)}(\mathbf{x})=b_{i}^{(\ell)}+\sum_{j_{\ell}=1}^{n_{\ell}}W_{ij_{\ell}}^{(\ell)}g_{j_{\ell}}^{(\ell)}(\mathbf{x}) \displaystyle g_{j_{\ell}}^{(\ell)}(\mathbf{x})=\sigma\left(z_{j_{\ell}}^{(\ell-1)}(\mathbf{x})\right) (1) \displaystyle j_{\ell}=1,2,\dots,n_{\ell},\quad g_{j}^{(0)}(\mathbf{x})=x_{j} Where • z_{i}^{(\ell)}(\mathbf{x}) : the i^{th} element of the output vector after \ell hidden layers with input \mathbf{x} 1 1 1 While in general these network have vector-valued outputs, this paper specifically addresses regression problems with one-dimensional outputs . • n_{0} : dimension of the input vector. • n_{\ell} : width of the \ell^{th} hidden layer. • W_{ij}^{(\ell)} : weight that associates node j in the current layer with node i of the next layer. The (\ell) indicates that this weight immediately proceeds the \ell^{th} hidden layer. • \sigma : a nonlinear activation function. In this paper, we use the ReLU activation function \sigma(x)=\max(0,x) . • x_{k} : the k^{th} element of the input vector \mathbf{x}\in\mathbb{R}^{n_{0}} . • b_{i}^{(\ell)} : bias term added after \ell hidden layers to form node i of the next layer. There is a modified version of the above setup that is necessary for neural tangent kernels: \displaystyle z_{i}^{\ell}(\mathbf{x})=b_{i}^{(\ell)}+\frac{1}{\sqrt{n_{\ell}}}\sum_{j_{\ell}=1}^{n_{\ell}}W_{ij_{\ell}}^{(\ell)}g_{j_{\ell}}^{(\ell)}(\mathbf{x}){} (2) The additional scaling factors \frac{1}{\sqrt{n_{\ell}}} are added because they allow for the use of the Weak Law of Large Numbers in the NTK derivation ( Jacot et al., 2018 ) . 2.2 Connection between Neural Networks and Gaussian Processes Neal (1996) showed that as the width of the hidden layer goes to infinity, the scalar output of pre-trained neural networks with one hidden layer (with independent and identically distributed (i.i.d.) parameter initialization) converges in distribution to a Gaussian process. This results require the following assumptions: (i) The weight and bias parameters are i.i.d. (ii) The width of hidden layers \to\infty . Lee

Read the complete text