Accelerating Neural Network Training Along Sharp and Flat Directions

Daniyar Zakarin, Sidak Pal Singh · 2025

Read Accelerating Neural Network Training Along Sharp and Flat Directions by Daniyar Zakarin, Sidak Pal Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11972.txt.

Introduction

Daniyar Zakarin and Sidak Pal Singh study how to speed neural network training in a 2025 paper. Recent work shows training steps align oddly with the sharpest directions of the loss surface. The pair test a variant of the standard training method that avoids those dominant sharp directions and steps only along flatter ones.

Stepping along flat directions sped convergence but could shake stability, and a few key settings governed the balance. Blended methods that mix the standard and variant steps unify the family and let users tune the trade-off. The team also ties the sharp subspace to known curvature terms, finding most curvature energy sits there. The work points toward curvature-aware training tools.

Catalog record

CREATORDaniyar Zakarin, Sidak Pal Singh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11972

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accelerating Neural Network Training Along Sharp and Flat Directions Daniyar Zakarin, Sidak Pal Singh Abstract Recent work has highlighted a surprising alignment between gradients and the top eigenspace of the Hessian—termed the Dominant subspace —during neural network training. Concurrently, there has been growing interest in the distinct roles of sharp and flat directions in the Hessian spectrum. In this work, we study Bulk -SGD, a variant of SGD that restricts updates to the orthogonal complement of the Dominant subspace. Through ablation studies, we characterize the stability properties of Bulk -SGD and identify critical hyperparameters that govern its behavior. We show that updates along the Bulk subspace , corresponding to flatter directions in the loss landscape, can accelerate convergence but may compromise stability. To balance these effects, we introduce interpolated gradient methods that unify SGD , Dom -SGD, and Bulk -SGD. Finally, we empirically connect this subspace decomposition to the Generalized Gauss–Newton and Functional Hessian terms, showing that curvature energy is largely concentrated in the Dominant subspace. Our findings suggest a principled approach to designing curvature-aware optimizers. 1 Introduction & Related Work Stochastic Gradient Descent (SGD) remains the cornerstone of optimization in deep learning. Any advancement that improves its convergence speed or ability to find better solutions is of substantial practical interest. A variety of first-order methods, including Momentum ( Sutskever et al., (2013) ) and Adam ( Kingma and Ba, (2017) ), have been proposed to address limitations of vanilla SGD. These algorithms are often motivated by insights into the geometry of neural network loss landscapes—particularly the need to efficiently traverse sharp and flat regions. Recent empirical investigations have illuminated a surprising structure in the Hessian of deep networks: it is typically dominated by a small number of large eigenvalues, suggesting a low-rank curvature profile Sagun et al., (2017) ; Sagun et al., (2018) ; Ghorbani et al., (2019) ; Papyan, (2019) . Gur-Ari et al., (2018) observed that the gradient vectors during training tend to align with the top-k eigenspace of the Hessian for k-class classification tasks. They hypothesized that learning predominantly occurs within this dominant subspace. However, this hypothesis was challenged in Song et al., (2025) , where authors performed controlled experiments to test its validity. Their results show that restricting updates to the dominant subspace fails to decrease the training loss beyond a certain point. In contrast, updates constrained to the orthogonal complement of this subspace achieve similar performance. This finding suggests that the observed alignment is not causally linked to effective optimization but is instead a byproduct of stochastic noise inherent in SGD. These insights offer both theoretical and practical implications. From a mechanistic standpoint, they suggest that much of the optimization trajectory may be spent moving in directions informed more by noise than by meaningful signal. This opens the door to the possibility that constraining updates to be orthogonal to high-curvature directions—effectively filtering out noisy components—could enable more stable and potentially larger steps. Such a strategy may accelerate convergence or help escape suboptimal basins by focusing descent along flatter directions in the landscape. 1.1 Description of the Main Results We summarize our main findings as follows: • In section 3.1 , we conduct a series of ablation studies to evaluate Bulk -SGD as a standalone training algorithm. We assess its sensitivity to key hyperparameters including the learning rate, the dimension of the Dominant subspace , and the Hessian holdout size . We analyze how the behaviors described in phenomena 2 and 3 vary when switching from standard SGD to Bulk -SGD at different training steps. Through these ablation studies identify and provide an explanation for an instability problem of Bulk -SGD. • In section 4 , we explore how the flatness of the Bulk subspace can be leveraged to accelerate neural network training. Through controlled experiments, we demonstrate that Bulk -SGD achieves faster convergence than classical SGD. In phenomenon 4 , we hypothesize distinct roles for the Dominant and Bulk components of the gradient—namely, that the Bulk component promotes acceleration, while the Dominant component contributes to stability. We further investigate the connection between these dynamics and generalization in phenomenon 5 . To address the trade-off between acceleration and stability, we introduce an interpolated gradient method that unifies SGD , Bulk -SGD, and Dom -SGD. We study how different interpolation regimes affect training loss and test accuracy. • In section 5 , we investigate the relationship between the Dominant and Bulk subspaces and the decomposition of the Hessian into the Generalized Gauss–Newton and Functional Hessian terms. We find that the energies of the full Hessian and the Generalized Gauss–Newton component align closely with the Dominant subspace , while the Functional Hessian exhibits little alignment. 2 Background Let \theta\in\mathbb{R}^{p} denote the parameters of a neural network trained with a twice-differentiable loss function \mathcal{L}:\mathbb{R}^{p}\rightarrow\mathbb{R} . Following the notation of Song et al., (2025) , we consider the spectrum of the Hessian \nabla^{2}\mathcal{L}(\theta) at a given point in the parameter space. Let \lambda_{1}(\theta)\geq\lambda_{2}(\theta)\geq\dots\geq\lambda_{p}(\theta) denote its ordered eigenvalues, with corresponding orthonormal eigenvectors u_{1}(\theta),u_{2}(\theta),\dots,u_{p}(\theta) . We define the Dominant subspace to be the span of the top- k eigenvectors: S_{k}(\theta)=\mathrm{Span}\left\{u_{1}(\theta),u_{2}(\theta),\dots,u_{k}(\theta)\right\}. In practice, k is typically chosen to match the number of output classes in a classification task. The orthogonal projector on this dominant subspace is denoted: P_{k}(\theta)\coloneqq\sum_{i=1}^{k}u_{i}(\theta)u_{i}(\theta)^{\top}. Let g=\nabla\mathcal{L}(\theta) denote the gradient of the loss. We decompose it into components aligned with the Dominant and complementary Bulk subspaces: g=P_{k}(\theta)g+P_{k}^{\perp}(\theta)g,\quad\text{where}\quad P_{k}^{\perp}(\theta)\coloneqq I-P_{k}(\theta). Gur-Ari et al., (2018) made a notable empirical observation about this decomposition: Phenomenon 1 . SGD gradients tend to align with the dominant subspace. Define the alignment coefficient as \chi_{k}(\theta)\coloneqq\frac{\|P_{k}(\theta)g\|_{2}}{\|g\|_{2}},\quad\text{where}\quad g=\nabla\mathcal{L}(\theta). They observe that, after a few initial steps of training with stochastic gradient descent (SGD), the alignment \chi_{k}(\theta_{t}) rapidly increases and stays near 1 throughout training. To test whether learning truly occurs along the dominant subspace, Song et al., (2025) propose two variants of gradient descent: • Dom -SGD: \theta_{t+1}\leftarrow\theta_{t}-\eta P_{k}(\theta_{t})g_{t} • Bulk -SGD: \theta_{t+1}\leftarrow\theta_{t}-\eta P_{k}^{\perp}(\theta_{t})g_{t} Here, g_{t}=\nabla\mathcal{L}(\theta_{t}) is the gradient at time t , and \eta is the learning rate. The following phenomena are observed: Phenomenon 2 . Dom-SGD is insufficient to reduce training loss. After training a model with standard SGD, switching to Dom-SGD halts further decrease in training loss. This suggests that, despite the apparent alignment, the Dominant subspace does not support continued learning. Phenomenon 3 . Bulk-SGD retains full training dynamics. When switching to Bulk-SGD, the training loss exhibits the same dynamics as the vanilla SGD. 3 Ablation Study 3.1 Bulk-SGD as a Standalone Optimizer \eta l k=1 k=5 k=10 k=20 SGD 0.01 200 39.59 39.09 38.98 37.70 39.67 0.01 500 39.55 39.00 38.86 38.40 0.01 5000 39.46 38.02 38.55 37.57 0.05 200 39.58 39.00 28.95 28.63 39.67 0.05 500 39.38 35.07 38.12 18.05 0.05 5000 39.60 38.45 38.38 35.88 0.1 200 30.49 40.71 40.02 19.92 40.84 0.1 500 41.12 39.74 29.74 29.40 0.1 5000 40.81 39.83 39.53 32.52 Table 1 : Test Accuracy on CIFAR10-5k using MLP (ReLU) with varying learning rates, Dominant subspace dimensions, and Hessian holdout sizes. The red values indicate the runs with the diverging loss. The best accuracy for each learning rate is indicated in bold. Table 2 : Test Accuracy on CIFAR10-5k using CNN with varying learning rates, dominant

Read the complete text