Zeroth-Order Sharpness-Aware Learning with Exponential Tilting

Xuchen Gong, Tian Li · 2025

Read Zeroth-Order Sharpness-Aware Learning with Exponential Tilting by Xuchen Gong, Tian Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16157.txt.

Introduction

Xuchen Gong and Tian Li join two lines of work on training neural networks. Both lines apply when gradients, the usual guide for learning, are off the table. One line tweaks the model's weights at random and keeps what lowers the average loss. The other, sharpness-aware training, hunts for the worst loss nearby so the model lands in a flat, safe valley.

Their bridge is an exponential tilting goal, a dial that slides smoothly between average loss and worst loss. On top of it they build training methods that need no gradients and little memory. Tests span image sorting, quiz answering, and text writing, and their methods beat plain no-gradient baselines. The work gives a cheaper path to the steady results that sharpness-aware training is prized for.

Catalog record

CREATORXuchen Gong, Tian Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16157

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zeroth-Order Sharpness-Aware Learning with Exponential Tilting Xuchen Gong, Tian Li Abstract Classic zeroth-order optimization approaches typically optimize for a smoothed version of the original function, i.e., the expected objective under randomly perturbed model parameters. This can be interpreted as encouraging the loss values in the perturbation set to be small on average. Popular sharpness-aware minimization (SAM) objectives, however, typically focus on the largest loss within the neighborhood to arrive at flat minima more effectively. In this work, we connect zeroth-order optimization (and its corresponding objectives) with SAM approaches explicitly, through an exponential tilting objective that provides a smooth transition between the average - and the max -loss formulations. We explore new zeroth-order algorithms to solve a soft SAM objective parameterized by a tilting parameter t t . We provide precise characterizations of the sharpness notions of the tilted SAM framework. Practically, our approach can be used as a gradient-free and memory-efficient alternative to SAM variants, and it achieves better generalization compared to vanilla zeroth-order baselines on a wide range of downstream tasks, including classification, multiple choice QA, and language generation. 1 Introduction Zeroth-order optimization has gained traction when the first-order or higher-order gradient access is unavailable, unreliable, or expensive. Applications include black-box adversarial attacks ( Chen et al. 2017 ) , fine-tuning large language models ( Chen et al. 2023 ; Malladi et al. 2023 ; Zhang et al. 2024 ) , differentially private learning ( Zhang et al. 2023 ; Tang et al. 2024 ) , and science problems. Consider the standard empirical risk minimization (ERM) problem: f(x)\coloneqq\frac{1}{N}\sum_{i=1}^{N}f(x;\xi_{i}) where x\in\mathbb{R}^{d} is the model parameters and \{\xi_{i}\}_{i\in[N]} represents training samples. One of the most popular zeroth-order algorithms relies on two function evaluations in the opposite directions to estimate the gradients. Such a two-point estimator takes the updating rule G(x,\rho,u)\coloneqq(1/2\rho)[f(x+\rho u)-f(x-\rho u)]u , where u is a random direction sampled from some distribution \mu(u) (e.g., uniform over a sphere or Gaussian), and \rho>0 is a smoothing parameter. Under mild assumptions, prior literature has shown that the two-point estimator optimizes an approximated, smooth version of the original function, i.e., \mathbb{E}_{u}[G(x,\rho,u)]=\nabla_{x}\mathbb{E}_{v}[f(x+\rho v)] 1 1 1 The distribution of v depends on that of u ; see Section 3 for details. ( Flaxman et al. 2004 ) . In other words, the zeroth-order method effectively minimizes the expected loss \mathbb{E}_{v}[f(x+\rho v)] in some perturbed neighborhood around x . Under such interpretation, zeroth-order optimization has a critical benefit that it is not originally designed for—ensuring the loss is small on average within the neighborhood so that the local minima can be flatter ( Wen et al. 2022 ; Tahmasebi et al. 2024 ; Zhang et al. 2025 ) . For over-parameterized and non-convex models, encouraging flatter local minima (i.e., optimizing a sharpness-aware objective) can be an effective technique that improves generalization performance ( Foret et al. 2020 ; Bahri et al. 2021 ) . However, the aforementioned vanilla zeroth-order estimate can only be viewed as a special sharpness-aware minimization (SAM) objective that focuses on the average loss. The canonical SAM approach and its variants typically uses a min-max formulation ( Foret et al. 2020 ) , which have been extensively studied in prior works and demonstrated strong empirical performance ( Wu et al. 2020 ; Sherborne et al. 2023 , e.g.,) . In this work, observing the connections between zeroth-order optimization and SAM, we explore the explicit bias of zeroth-order optimization towards flat solutions in detail. We develop new zeroth-order algorithms that solve a continuous spectrum of sharpness-aware objectives, ranging from the average- to the max-loss formulations, leveraging exponential tilting. Exponential tilting has been used as a common technique to create parametric distribution shifts in various contexts ( Dembo 2009 ; Li et al. 2023 ; Robey et al. 2022 ) . It has also been used to develop new sharpness-aware objectives that reweigh different local minima ( Li et al. 2024 ) . Our zeroth-order algorithms solve a similar objective to arrive at flatter solutions, while preserving the same computational and memory efficiency as the classic zeroth-order methods. To be more specific, we consider a soft SAM objective parameterized by a tilting parameter t (named tilted SAM or t -SAM ( Li et al. 2024 ) ) that covers the average and min-max formulation as special cases. To approximate the unbiased gradient estimator of the titled SAM objective, we propose different strategies based on finite function evaluations under random perturbations of the model parameter. Additionally, we provide the precise characterizations of a family of sharpness notions of the tilted SAM framework and the solutions it favors, as a function of the tilting parameter t . While our framework in principle applies to any form of model perturbations, we investigate the cases with Gaussian and ball-constrained uniform perturbations in detail. Our z eroth-order e xponential-tilted s harpness-aware t raining (ZEST) approach achieves superior performance compared with vanilla two-point estimator (corresponding to solving an average-loss based sharpness-aware objective) across various model types and downstream tasks, including classification, multiple choice QA, and language generation (Section 5 ). In applications where zeroth-order optimization is competitive in general, ZEST can even achieve higher accuracies than first-order SAM variants while being gradient-free and memory-efficient (Section 5.2 ). In summary, our contributions are as follows. In Section 3 , we propose a new zeroth-order optimization algorithm (ZEST) that uses exponential tilting to recover a smooth spectrum of sharpness-aware objectives. Theoretically , we analyze the explicit bias (the “sharpness” notion) of the t -SAM objective and illustrate how ZEST can reach flatter minima than the baselines. We show that our method can identify and conservatively avoid minima with large curvatures in any direction, while vanilla zeroth-order methods cannot (Section 4 ). Empirically , in Section 5 , we evaluate ZEST on comprehensive language tasks and different model types, demonstrating that ZEST performs better than zeroth-order baselines while being equally fast and memory-efficient. 2 Preliminaries and Related Work Zeroth-Order Optimization. Zeroth-order methods ( Spall 2002 ; Shamir 2017 ; Bach & Perchet 2016 ; Duchi et al. 2015 ; Jamieson et al. 2012 ; Liu et al. 2018 ; Nesterov & Spokoiny 2017 ; Agarwal et al. 2011 ) have gained recent attention due to their promising performance in fine-tuning language models and their memory efficiency, at the cost of increased iteration complexity compared to first-order methods ( Malladi et al. 2023 ; Zhang et al. 2023 ) . Zeroth-order methods typically optimize for a smoothed version of the functions, which can be interpreted as the expected loss values under perturbed model parameters. Enforcing that the loss values are small in expectation has connections with a special case of sharpness-aware approaches ( Zhang et al. 2025 ) , where sharpness is defined as the trace of the Hessian ( Wen et al. 2022 ) . Specifically, by Taylor expansion, the effective objective (proved in Appendix A ) is \displaystyle\mathbb{E}_{v}[f(x+\rho v)]=\underbrace{f(x)}_{\text{Empirical loss}}+\underbrace{\frac{\rho^{2}}{2}\text{Tr}(\nabla^{2}f(x))}_{\text{Sharpness }R_{\text{avg}}}+O(\rho^{2}d). In this work, we develop new zeroth-order algorithms that solve a spectrum of SAM objectives that cover this special case (Section 3 ) and provide precise characterizations of sharpness in our approach (Section 4 ). Sharpness-Aware Minimization. Sharpness-Aware Minimization (SAM) and its variants have been extensively studied in prior work (

Read the complete text