Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability
2025
Read Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12837.txt.
Introduction
Kaiqi Jiang, Jeremy Cohen, and Yuanzhi Li study how a training diagnostic evolves at the edge of stability in a 2025 paper. The catalog record lists no creator; the byline names the three authors. The neural tangent kernel tracks how a network's predictions shift during training, and its largest eigenvalue hovers near a stability limit when step sizes are large.
What its eigenvectors do in that regime was unknown. Across architectures, the team found that bigger learning rates rotate the final kernel's top directions toward the training target, aligning the whole kernel more with the task. A two-layer linear model yielded a full theoretical account of the effect. The work sharpens the picture of how aggressive training tunes feature learning itself.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12837
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability Abstract The study of Neural Tangent Kernels (NTKs) in deep learning has drawn increasing attention in recent years. NTKs typically actively change during training and are related to feature learning. In parallel, recent work on Gradient Descent (GD) has found a phenomenon called Edge of Stability (EoS), in which the largest eigenvalue of the NTK oscillates around a value inversely proportional to the step size. However, although follow-up works have explored the underlying mechanism of such eigenvalue behavior in depth, the understanding of the behavior of the NTK eigenvectors during EoS is still missing. This paper examines the dynamics of NTK eigenvectors during EoS in detail. Across different architectures, we observe that larger learning rates cause the leading eigenvectors of the final NTK, as well as the full NTK matrix, to have greater alignment with the training target. We then study the underlying mechanism of this phenomenon and provide a theoretical analysis for a two-layer linear network. Our study enhances the understanding of GD training dynamics in deep learning. Kaiqi Jiang {}^{\;\dagger} Jeremy Cohen {}^{\;\ddagger} Yuanzhi Li {}^{\;\ddagger} Department of Electrical and Computer Engineering, Princeton University {}^{\;\dagger} Machine Learning Department, Carnegie Mellon University {}^{\;\ddagger} August 8, 2026 1 Introduction Gradient Descent (GD) is a canonical minimization algorithm widely used in optimization and machine learning problems. While its behavior and guarantee in convex settings has been fully studied, there are still enormous challenges when understanding its dynamics and convergence in non-convex settings, especially for deep learning problems where we usually encounter high-dimensional and highly non-convex loss functions. In recent years, the study on Neural Tangent Kernel (NTK) proposed in [ 20 ] has been more and more popular when understanding the learning mechanism of GD in deep neural networks, for example, [ 14 ] first proved the convergence guarantee of GD in the so-called lazy regime, where NTK remains unchanged during training. More and more follow-up studies point out the limitation of this lazy regime and advocate for the analysis in the rich regime where the NTK actively evolves during training. They argue that neural networks can learn more efficiently than kernel methods [ 16 , 33 , 12 ] . These results beyond the NTK story motivate us to track the learning dynamics and the evolution of NTK more carefully when training neural networks. There has also been a line of research on the behavior of NTK eigenvalues when training neural networks using GD. [ 10 ] observed a phenomenon called Edge of Stability (EoS) where the largest eigenvalue of loss Hessian first increases and then hovers around a value inversely proportional to the step size during training. This can also translate to similar behavior of the largest eigenvalue of NTK if we use MSE loss. There is a lot of follow-up work studying the underlying mechanism of this eigenvalue behavior (e.g. [ 34 , 13 ] ). Despite the success of those works in studying the behavior of eigenvalues of NTK, the understanding of the rotation of its eigenvectors, especially those leading ones, is still lacking. Inspired by the following evolution equation under Gradient Flow (GF), \frac{d}{dt}(f_{t}-y)=-K_{t}(f_{t}-y) where f_{t} is the model output at time t , y is the target vector, and K_{t} represents NTK at time t , we are interested in studying the alignment between NTK and the target y and how this alignment evolves over time. The study of the alignment of NTK and the target is not a new story [ 23 , 29 , 27 , 32 ] . People have developed different metrics to measure the alignment, e.g. Kernel Target Alignment (KTA) [ 11 ] . Definition 1 (Full Kernel Target Alignment (KTA)) . Let y\in\mathbb{R}^{m} be the target vector and K\in\mathbb{R}^{m\times m} be the NTK. Then we say that the kernel target alignment is \frac{y^{T}Ky}{\|y\|_{2}^{2}\|K\|_{F}} . However, there has been a lack of research on how the EoS phenomenon interacts with the evolution of the NTK eigenvectors and the kernel target alignment. This motivates us to study the following question: How do eigenvectors of NTK evolve over time and align with the target, especially during EoS when the NTK eigenvalues exhibit rich behavior? 1.1 Overview of Our Main Contributions In addition to the KTA in Definition 1 , we are also interested in the individual eigenvector target alignment : Definition 2 (Individual Eigenvector Target Alignment) . For the target vector y\in\mathbb{R}^{m} , if u_{k}\in\mathbb{R}^{m} is the k -th eigenvector of the NTK, then we say that its alignment with the target is \frac{(y^{T}u_{k})^{2}}{\|y\|_{2}^{2}} . Note that the alignment values sum up to 1, so they can be regarded as a kind of “distribution”. We aim to track the evolution of these two metrics during GD training and study their connection to EoS. We make the following contributions: 1. Across different architectures, we observe that larger GD learning rates cause the final NTK matrix to have higher KTA alignment (Definition 1 ) with the target, as shown in Figure 1(a) . Looking into the mechanism, we observe that larger learning rates cause the leading eigenvectors of the final NTK to have greater individual alignment with the target (Definition 2 ). Namely, larger learning rates cause the “distribution” of the Individual Eigenvector Target Alignment to shift towards leading eigenvectors, as shown in Figure 1(b) . We refer to this phenomenon as alignment shift . 2. We study the underlying mechanism of the alignment shift phenomenon by tracking the detailed training dynamics. We find that during the phases of EoS when the sharpness reduces, we usually observe sudden or fast increases in alignment, as shown in Figure 1(c) . 3. To better understand these empirical findings, we theoretically analyze the alignment shift phenomenon in a 2-layer linear network. This analysis reveals that the phase of EoS when the sharpness reduces is not only correlated with but essentially a contributing factor to the alignment shift. We also partially generalize our theoretical analysis to nonlinear networks in Appendix A.6 . 4. To further link sharpness reduction with increased kernel-target alignment, we use the central flow framework [ 8 ] , which models gradient descent at EoS as implicitly following a sharpness-penalized gradient flow. We show empirically that this sharpness penalty leads to increased KTA. (a) (b) (c) Figure 1 : (a) KTA across learning rates, (b) the alignment shift, and (c) connection between the alignment dynamics and Edge of Stability in a fully-connected network (see Section 3.1 ). Each learning rate was trained until reaching roughly the same loss value. Column (c) singles out the trajectory with the learning rate 0.1. Note that (b) only plots the alignment on top 20 eigenvectors as later ones have much smaller alignment. The NTK eigenvalues are normalized by dividing by the \ell_{2} norm of the eigenvalues, i.e. the Froebenius norm of the NTK. 2 Related Work Study on eigenvalues of NTK [ 24 ] showed that when wide neural networks are trained with a learning rate large enough to trigger initial instability, the largest eigenvalue of the NTK drops until stability is restored, an effect referred to as the “catapult mechanism.” [ 10 ] showed that