Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training
Yue Hu, Zanxia Cao, Yingchao Liu · 2025
Read Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training by Yue Hu, Zanxia Cao, Yingchao Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19968.txt.
Introduction
Yue Hu, Zanxia Cao, and Yingchao Liu proposed a new training method for neural networks in a 2025 paper. Standard tools such as SGD and Adam use only slope data, so they can stall on flat spots and saddle points. Tools that use curve data work better there but cost too much for large models.
Their method, called Dimer-Enhanced Optimization, borrows a trick from studies of molecules. It places two close points on the loss surface to sense the curve with slope data alone. The method then steers the training steps away from saddle points and flat ground. Early tests on a small Transformer model matched the standard tools, and the authors present the method as a cheap way to cross hard loss surfaces.
Catalog record
| CREATOR | Yue Hu, Zanxia Cao, Yingchao Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19968
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training Yue Hu, Zanxia Cao, Yingchao Liu Abstract First-order optimization methods, such as SGD and Adam, are widely used for training large-scale deep neural networks due to their computational efficiency and robust performance. However, relying solely on gradient information, these methods often struggle to navigate complex loss landscapes featuring flat regions, plateaus, and saddle points. Second-order methods, which utilize curvature information from the Hessian matrix, can address these challenges but are computationally infeasible for large models. The Dimer method, a first-order technique that constructs two closely spaced points to probe the local geometry of a potential energy surface, offers an efficient approach to estimate curvature using only gradient information. Drawing inspiration from its application in molecular dynamics simulations for locating saddle points, we propose Dimer-Enhanced Optimization (DEO), a novel framework to facilitate escaping saddle points in neural network training. Unlike its original use, DEO adapts the Dimer method to explore a broader region of the loss landscape, efficiently approximating the Hessian’s smallest eigenvector without computing the full Hessian matrix. By periodically projecting the gradient onto the subspace orthogonal to the minimum curvature direction, DEO guides the optimizer away from saddle points and flat regions, promoting more effective training while significantly reducing sampling time costs through non-stepwise updates. Preliminary experiments on Transformer-based toy models show that DEO achieves competitive performance compared to standard first-order methods, enabling improved navigation of complex loss landscapes. Our work offers a practical approach to repurpose physics-inspired, first-order curvature estimation for enhancing neural network training in high-dimensional spaces. 1 Introduction Training deep neural networks involves optimizing high-dimensional, non-convex loss functions, often hindered by complex loss landscapes featuring flat regions, narrow valleys, and saddle points [ 1 ] . First-order methods, such as Stochastic Gradient Descent (SGD) [ 2 ] , Adam [ 3 ] , and AdamW [ 4 ] , are widely adopted for their computational efficiency and robust performance. These methods rely on local gradient information, leading to slow convergence in ill-conditioned regions or stagnation near saddle points where the gradient is near zero but not at a local minimum. Second-order methods leverage the Hessian matrix’s curvature information via the Newton step \Delta\mathbf{\theta}=-\mathbf{H}^{-1}\mathbf{g} , but their \mathcal{O}(N^{2}) memory and \mathcal{O}(N^{3}) complexity make them infeasible for large models. Recent methods like Sophia [ 5 ] approximate the Hessian’s diagonal with gradient differences, achieving low overhead but missing non-diagonal interactions critical for saddle points. Inspired by the Dimer method from molecular dynamics [ 6 ] , which probes potential energy surfaces to locate saddle points, we propose Dimer-Enhanced Optimization (DEO). DEO adapts the Dimer method to escape saddle points and stabilize neural network training, using a displacement \Delta R\approx 10\cdot\text{lr} to probe broader loss landscapes and periodically correcting gradients of optimizers like Adam. Experiments on Transformer-based models demonstrate DEO’s competitive performance and its ability to mitigate training instabilities, offering a practical approach for more robust optimization. 2 Background and Related Work DEO integrates first-order optimization, curvature estimation, and saddle point escape strategies. 2.1 First-Order Stochastic Optimizers SGD [ 2 ] updates parameters using mini-batch gradients but converges slowly in complex landscapes. Adam [ 3 ] combines momentum and adaptive learning rates, while AdamW [ 4 ] improves generalization with decoupled weight decay. Sophia [ 5 ] approximates the Hessian’s diagonal with gradient differences, achieving low overhead. These methods (SGD, Adam, AdamW, Sophia) are our baselines. 2.2 Our Contribution DEO enhances first-order optimizers by: - Using the Dimer method for non-diagonal curvature, unlike gradient-only SGD, Adam, AdamW. - Targeting the Hessian’s smallest eigenvector, unlike Sophia’s diagonal focus. - Employing \Delta R\approx 10\cdot\text{lr} and periodic updates for efficiency. 3 Methodology We develop Dimer-Enhanced Optimization (DEO) by adapting the Dimer method [ 6 ] to escape saddle points, aligning with its molecular dynamics formulation. 3.1 Preliminaries: The Dimer Method The Dimer method locates saddle points in molecular dynamics by probing potential energy surfaces [ 6 ] . For a loss function L(\mathbf{\theta}) , DEO adapts this to estimate the Hessian’s smallest eigenvector \hat{\mathbf{N}} using gradients. We construct one additional point with \Delta R\approx 10\cdot\text{lr} (e.g., \Delta R=6\times 10^{-3} for \text{lr}=6\times 10^{-4} ) to probe a broader loss landscape, using the current point’s gradient. Given \mathbf{\theta} and \mathbf{g}=\nabla L(\mathbf{\theta}) , we construct: \mathbf{\theta}_{2}=\mathbf{\theta}+\Delta R\hat{\mathbf{N}} with gradient \mathbf{g}_{2}=\nabla L(\mathbf{\theta}_{2}) . The rotational force updates \hat{\mathbf{N}} : \mathbf{F}_{R}=(\mathbf{g}_{2}-\mathbf{g})-\big((\mathbf{g}_{2}-\mathbf{g})\cdot\hat{\mathbf{N}}\big)\hat{\mathbf{N}} \hat{\mathbf{N}}_{t+1}=\text{normalize}(\hat{\mathbf{N}}_{t}+\eta_{\text{rot}}\mathbf{F}_{R}) where \eta_{\text{rot}}=10^{-3} . The curvature is: C=\frac{L(\mathbf{\theta}_{2})-L(\mathbf{\theta})}{\Delta R} A negative C confirms alignment with the minimum curvature direction, requiring one extra gradient evaluation [ 6 ] . 3.2 Gradient Correction DEO corrects the gradient by projecting it orthogonal to \hat{\mathbf{N}} , with coefficient \alpha (e.g., 5.0): \mathbf{g}_{\text{mod}}=\mathbf{g}-\alpha(\mathbf{g}\cdot\hat{\mathbf{N}})\hat{\mathbf{N}} This removes low-curvature components, guiding the optimizer away from saddle points. 3.3 Dimer-Enhanced Optimization (DEO) with Adam DEO enhances the Adam optimizer [ 3 ] with periodic Dimer corrections. Adam updates parameters using: \mathbf{m}_{t}=\beta_{1}\mathbf{m}_{t-1}+(1-\beta_{1})\mathbf{g}_{t},\quad\mathbf{v}_{t}=\beta_{2}\mathbf{v}_{t-1}+(1-\beta_{2})\mathbf{g}_{t}^{2} \hat{\mathbf{m}}_{t}=\frac{\mathbf{m}_{t}}{1-\beta_{1}^{t}},\quad\hat{\mathbf{v}}_{t}=\frac{\mathbf{v}_{t}}{1-\beta_{2}^{t}} \mathbf{\theta}_{t+1}=\mathbf{\theta}_{t}-\text{lr}\cdot\frac{\hat{\mathbf{m}}_{t}}{\sqrt{\hat{\mathbf{v}}_{t}}+\epsilon} where \beta_{1}=0.9 , \beta_{2}=0.95 , \epsilon=10^{-8} . DEO replaces \mathbf{g}_{t} with \mathbf{g}_{\text{mod}} from the Dimer correction, using \Delta R\approx 10\cdot\text{lr} . Algorithm 1 details the integration. Algorithm 1 Dimer-Enhanced Optimization (DEO) with Adam 1: Input : Loss function L(\mathbf{\theta}) , learning rate lr , update frequency f , displacement \Delta R\approx 10\cdot\text{lr} , step size \eta_{\text{rot}} , correction coefficient \alpha , Adam parameters \beta_{1} , \beta_{2} , \epsilon 2: Initialize \mathbf{\theta} , \hat{\mathbf{N}}_{\text{cached}}\leftarrow random unit vector, \mathbf{m}_{0}\leftarrow\mathbf{0} , \mathbf{v}_{0}\leftarrow\mathbf{0} 3: for each step t do 4: Compute gradient \mathbf{g}\leftarrow\nabla L(\mathbf{\theta}) 5: if t\mod f=0 then \triangleright Expensive step 6: \mathbf{\theta}_{2}\leftarrow\mathbf{\theta}+\Delta R\hat{\mathbf{N}}_{\text{cached}} 7: \mathbf{g}_{2}\leftarrow\nabla L(\mathbf{\theta}_{2}) 8: \mathbf{F}_{R}\leftarrow(\mathbf{g}_{2}-\mathbf{g})-\big((\mathbf{g}_{2}-\mathbf{g})\cdot\hat{\mathbf{N}}_{\text{cached}}\big)\hat{\mathbf{N}}_{\text{cached}} 9: \hat{\mathbf{N}}_{\text{new}}\leftarrow\text{normalize}(\hat{\mathbf{N}}_{\text{cached}}+\eta_{\text{rot}}\mathbf{F}_{R}) 10: \hat{\mathbf{N}}_{\text{cached}}\leftarrow\hat{\mathbf{N}}_{\text{new}} 11: \mathbf{g}_{\text{mod}}\leftarrow\mathbf{g}-\alpha(\mathbf{g}\cdot\hat{\mathbf{N}}_{\text{new}})\hat{\mathbf{N}}_{\text{new}} 12: else \triangleright Cheap step 13: \mathbf{g}_{\text{mod}}\leftarrow\mathbf{g}-\alpha(\mathbf{g}\cdot\hat{\mathbf{N}}_{\text{cached}})\hat{\mathbf{N}}_{\text{cached}} 14: end if 15: \mathbf{m}_{t}\leftarrow\beta_{1}\mathbf{m}_{t-1}+(1-\beta_{1})\mathbf{g}_{\text{mod}} 16: \mathbf{v}_{t}\leftarrow\beta_{2}\mathbf{v}_{t-1}+(1-\beta_{2})\mathbf{g}_{\text{mod}}^{2} 17: \hat{\mathbf{m}}_{t}\leftarrow\mathbf{m}_{t}/(1-\beta_{1}^{t}) , \hat{\mathbf{v}}_{t}\leftarrow\mathbf{v}_{t}/(1-\beta_{2}^{t}) 18: \mathbf{\theta}_{t+1}\leftarrow\mathbf{\theta}_{t}-\text{lr}\cdot\hat{\mathbf{m}}_{t}/(\sqrt{\hat{\mathbf{v}}_{t}}+\epsilon) 19: end for 4 Experiments We evaluate DEO using two Transformer-based toy models of varying complexity, inspired by the nanoGPT framework [ 7 ] . The experiments are conducted on a large public text corpus and compare DEO-enhanced optimizers against their standard counterparts. 4.1 Experiment 1: Simpler Language Model 4.1.1 Experimental Setup • Model : A simpler language model with 6 Transformer layers, 4 attention heads, and a hidden dimension of 256. The model is configured for sequence classification. • Dataset : A large public text corpus, with a micro-batch size of 12 and 20 gradient accumulation steps (total batch size of 240). • Optimizers : DEO (with Adam, AdamW, SGD, and Sophia bases), Adam, AdamW, SGD, and Sophia. • Hyperparameters : Learning rate \text{lr}=6\times 10^{-4} with cosine decay, \Delta R=6\times 10^{-3} , f=10 , and \alpha=5.0 . 4.1.2 Results As shown in Figure 1 , the DEO-enhanced optimizers performed competitively. The loss curves for DEO variants closely tracked or slightly outperformed their baselines, particularly the Adam and AdamW bases. While SGD-based optimizers converged more slowly, the Dimer-enhanced version achieved a slightly lower final loss, indicating a consistent, albeit small, benefit from the gradient correction. Sophia also performed strongly, but the results show that DEO provides a comparable level of performance in this setting. Figure 1: Raw (unsmoothed) training loss comparison on a simpler language toy model. DEO-enhanced variants show competitive performance, particularly with Adam and AdamW bases. 4.2 Experiment 2: More Complex Language Model 4.2.1 Experimental Setup • Model : A more complex language model with