Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization

Ruoran Xu, Borong She, Xiaobo Jin, Qiufeng Wang · 2026

Read Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization by Ruoran Xu, Borong She, Xiaobo Jin, Qiufeng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29547.txt.

Introduction

Ruoran Xu and three colleagues tackle non-smooth optimization, where rectified linear units and quantization break the smooth-loss premise. Adaptive methods like Adam then suffer gradient chattering, violent swings from conflicting Clarke subdifferential signals. Their Singularity-aware Adam damps step sizes by local geometric instability, via a new LGI metric from randomized directional derivatives. The method converges to Clarke stationary points at the optimal rate. Tests on quantization-aware training and noisy small-batch learning beat AdamW and Prox-SGD, with gains above four percent on CIFAR-100 and TinyImageNet.

Catalog record

CREATORRuoran Xu, Borong She, Xiaobo Jin, Qiufeng Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64990
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29547

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization Ruoran Xu, Borong She, Xiaobo Jin, Qiufeng Wang Abstract Deep learning optimization relies heavily on the assumption of smooth loss landscapes, a condition systematically violated by modern architectures due to non-smooth components like ReLU activations and quantization operators. In such non-smooth regimes, adaptive optimizers such as Adam suffer from gradient chattering —violent oscillations caused by conflicting signals within the Clarke subdifferential—leading to poor convergence and suboptimal generalization. To address this, we introduce Singularity-aware Adam (S-Adam) , a novel optimizer that stabilizes training by dynamically modulating step sizes based on local geometric instability. Our key contribution is the Local Geometric Instability (LGI) metric, a computationally efficient estimator of the Clarke subdifferential diameter derived from the variance of randomized directional derivatives. S-Adam incorporates an adaptive damping mechanism exp ⁡ ( − λ ​ ρ t ) \exp(-\lambda\rho_{t}) that decelerates updates in high-instability regions while preserving fast convergence in smooth basins. We provide a rigorous convergence analysis using differential inclusions, proving that S-Adam converges almost surely to ( δ , ϵ ) (\delta,\epsilon) -Clarke stationary points at the optimal 𝒪 ⁡ ( 1 / T ) \mathcal{O}(1/\sqrt{T}) rate. Empirical evaluations on Quantization-Aware Training (QAT) and high-noise small-batch learning demonstrate that S-Adam consistently outperforms AdamW and Prox-SGD, achieving accuracy gains of up to +6% on CIFAR-100 and +3% on TinyImageNet while effectively mitigating gradient oscillations. 1 Introduction The theoretical underpinnings of deep learning optimization are predominantly built upon the assumption of Lipschitz-continuous gradients, which guarantees stable descent and convergence. However, this assumption is fundamentally incompatible with the architectural realities of modern neural networks, which ubiquitously incorporate non-smooth elements such as ReLU activations, quantization functions in Quantization-Aware Training (QAT) ( 17 ) , and sparsity-inducing regularizers. While these functions are locally Lipschitz almost everywhere, they introduce sharp singularities where gradients are undefined, rendering classical smooth optimization theory inadequate. (a) Global minimum point and ideal trajectory (b) Comparison of Adam and Prox-SGD trajectories (c) S-Adam: LGI-triggered damping (d) Stabilized convergence of S-Adam Figure 1 : Geometric instability visualization on synthetic non-smooth landscape of f(x,y)=|x-1|+|y-1|+0.5(x^{2}+y^{2}) Figure 2 : Figure 1 (d) on synthetic non-smooth landscape At such non-smooth points, the local geometry is characterized not by a single gradient vector but by the Clarke subdifferential \partial_{C}f(x) —a convex set encompassing all possible limiting gradients ( 6 ) . When the diameter of this set is large, momentum-based optimizers like Adam ( 18 ) accumulate conflicting directional signals, leading to gradient chattering : pathological oscillations that prevent convergence into sharp, low-loss basins and degrade generalization performance. Existing remedies fall short in practice. Proximal methods, while theoretically sound, require explicit proximal mappings that are intractable for nested non-smooth functions like those in QNNs (Quantized Neural Networks). Randomized smoothing techniques, conversely, introduce prohibitive variance when used as gradient surrogates in large-scale training. Crucially, both approaches lack a mechanism to dynamically adapt to local geometric instability without compromising computational efficiency. In this work, we propose a paradigm shift: instead of smoothing the objective, we employ randomized smoothing as a geometric probe to estimate the local instability of the loss landscape. This insight leads to our main contribution, Singularity-aware Adam (S-Adam) (In Fig. 1 and Fig. 2 ), which introduces: • The Local Geometric Instability (LGI) metric, a variance-based estimator of the Clarke subdifferential diameter that requires no Hessian computation; • An adaptive geometric brake \exp(-\lambda\rho_{t}) that modulates step sizes in real-time, dampening updates near singularities while maintaining rapid progress in smooth regions; • A rigorous convergence guarantee to Clarke stationary points with an optimal \mathcal{O}(1/\sqrt{T}) rate, established via differential inclusion analysis; • Empirical superiority in extreme non-smooth settings, including low-bit QAT and high-noise small-batch learning, where S-Adam achieves significant accuracy improvements over AdamW and Prox-SGD. Our work bridges the gap between non-smooth optimization theory and practical deep learning scalability, offering a drop-in replacement for Adam that is both theoretically grounded and empirically robust. 2 Related Work Our work sits at the intersection of non-smooth optimization, adaptive gradient methods, and geometric-aware learning. We contextualize S-Adam within three key strands of literature. 2.1 Non-smooth Optimization and Proximal Methods Classical non-smooth optimization relies on subgradient methods ( 27 ) , which converge weakly to critical points at sublinear rates. Proximal algorithms ( 24 ; 29 ) offer stronger guarantees by solving implicit subproblems via the proximal operator. However, these methods require the non-smooth term to be proximable —admitting a closed-form or efficiently computable proximal mapping. This assumption fails in modern deep learning, where non-smoothness arises from nested compositions of rounding functions (quantization), ReLUs, and sparsity-inducing penalties. The lack of tractable proximal mappings for such composite operators renders standard proximal methods computationally prohibitive or inapplicable ( 9 ) . 2.2 Smoothing Techniques and Gradient Estimation Smoothing methods approximate non-smooth objectives with differentiable surrogates. Nesterov smoothing ( 23 ) convolves the objective with a strongly convex regularizer, while randomized smoothing ( 12 ) uses expectation over random perturbations. Although effective in theory, using the gradient of the smoothed function introduces estimation bias and high variance that degrades performance in high-precision training ( 5 ) . More critically, these approaches replace the original gradient, altering the optimization dynamics and potentially converging to different minima. S-Adam takes a fundamentally different stance: we employ randomized smoothing not as a gradient surrogate, but as a diagnostic probe to estimate local geometric instability while preserving the original gradient direction. 2.3 Adaptive Optimizers and Stability Adaptive gradient methods like Adam ( 18 ) and its variants (AdamW ( 21 ) , AdaBound ( 22 ) ) scale learning rates using exponential moving averages of past gradients. These methods implicitly assume local smoothness to trust their second-moment estimators \hat{v}_{t} . At non-smooth singularities, instantaneous gradient jumps violate this assumption, causing \hat{v}_{t} to lag behind the rapidly changing geometry—a mismatch that triggers gradient chattering. Recent attempts to stabilize adaptive optimizers include gradient clipping ( 25 ) , learning rate warmup ( 14 ) , and variance reduction ( 10 ) . However, these heuristics lack a principled mechanism to distinguish between stochastic noise and genuine topological singularities. 2.4 Geometry-Aware Minimization Sharpness-Aware Minimization (SAM) ( 13 ) seeks flat minima by minimizing the worst-case loss within a neighborhood. While effective for generalization, SAM relies on a first-order Taylor expansion to approximate the perturbation direction—an approximation that is theoretically invalid at non-differentiable points where the gradient is undefined or discontinuous. Consequently, SAM struggles to characterize sharpness accurately in quantized or sparsely regularized landscapes. Subsequent variants like ASAM ( 19 ) attempt to address scale invariance but retain the fundamental limitation at singularities. Other geometry-aware approaches include trust-region methods ( 7 ) and second-order preconditioning ( 1 ) , but these typically require explicit Hessian information or are limited to convex settings. 2.5 Positioning of S-Adam S-Adam distinguishes itself by: (1) providing a computationally lightweight geometric probe (LGI) that requires no Hessian computation, (2) maintaining the original gradient direction while adaptively modulating step sizes, (3) offering rigorous convergence guarantees to Clarke stationary points in non-smooth regimes, and (4) seamlessly degenerating to standard Adam in smooth regions. Unlike proximal methods, S-Adam handles nested non-smoothness; unlike smoothing techniques, it preserves gradient fidelity; and unlike SAM, it remains valid at

Read the complete text