CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning

Shuo Wang, Ziyu Chen, Ming Tang · 2026

Read CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning by Shuo Wang, Ziyu Chen, Ming Tang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21725.txt.

Introduction

Wang and colleagues propose CurvZO, a sparse zeroth-order method for fine-tuning language models with less memory. Such updates use only forward passes, saving memory, but they can converge slowly or prove unstable. CurvZO tracks curvature signals online from scalar feedback, uses them to pick which parameters to perturb each step, and adjusts its perturbation budget over time, cutting variance. Tests on OPT and Llama across varied tasks found gains of up to 4.4 accuracy points and up to twice the speed.

Catalog record

CREATORShuo Wang, Ziyu Chen, Ming Tang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76562
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21725

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning Shuo Wang, Ziyu Chen, Ming Tang Abstract Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effectiveness hinges on selecting informative parameters, which is challenging in ZO optimization because each query yields only scalar feedback. We propose Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO) , which tracks curvature signals online from scalar ZO feedback and leverages these signals to construct a parameter-wise sampling distribution for selecting coordinates at each update, reducing the variance of the sparse ZO gradient estimator. Moreover, CurvZO dynamically adapts the perturbation budget to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. Extensive experiments on OPT and Llama across diverse NLP tasks show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines. It improves accuracy by up to 4.4 points and achieves up to a 2 × 2\times speedup, while preserving memory efficiency. 1 Introduction Fine-tuning large language models (LLMs) via backpropagation is a widely adopted strategy that achieves strong performance across a variety of tasks. However, it imposes significant memory overhead due to the large number of parameters, which limits its scalability on resource-constrained hardware. As model sizes continue to grow rapidly, often by orders of magnitude every few years, the gap between memory requirements and available hardware capacity becomes increasingly pronounced ( 11 ; 14 ) . This trend intensifies the so-called memory wall ( 8 ) and poses challenges for both training and deployment ( 31 ; 13 ) . To address these limitations, zeroth-order (ZO) optimization has emerged as a memory-efficient alternative for fine-tuning LLMs ( 25 ; 17 ; 34 ; 33 ) . ZO methods estimate gradients using only forward passes, avoiding backpropagation and largely eliminating the memory cost of storing activations, gradients, and optimizer states. Among them, MeZO ( 18 ) further reduces memory by re-generating perturbations from a saved random seed, so training memory becomes close to inference, yielding up to 12\times savings in practice. Figure 1 : Visualization of anisotropic local curvature in the attention output weights of OPT-6.7B. The x - and y -axes index the columns and rows of the weight matrix, while the z -axis shows curvature magnitude approximated via the diagonal Fisher information (a standard local-curvature surrogate in neural networks). However, ZO optimization often suffers from unstable convergence and lower accuracy compared to first-order (FO) methods that rely on backpropagation for gradient computation, largely due to noise in ZO gradient estimates ( 17 ) . Prior studies have indicated that this noise scales with the dimensionality of the optimized parameters, leading to further degradation as dimensionality increases ( 17 ; 30 ; 10 ) . Although prior works attempt to mitigate this issue by reducing the number of parameters involved in ZO fine-tuning, they do not provide a principled way to identify which subset of parameters should be perturbed. This limitation arises because the accessible information in the ZO setting is fundamentally restricted to scalar responses from random perturbations. To address this, existing methods incorporate additional information or pre-computed statistics. For instance, SensZOQ ( 10 ) constructs a sparsity pattern using the empirical Fisher information matrix, computed offline before fine-tuning. Such reliance on additional information or pre-computed statistics introduces non-negligible computational overhead, thereby undermining the primary advantage of ZO methods: memory efficiency and simplicity. This naturally leads to the following question: Can we design a plug-and-play mechanism that selectively allocates perturbations to the most informative parameters, thereby improving ZO fine-tuning without relying on pre-computed statistics or changes to the existing framework? Recent studies have shown that the loss landscape of LLMs can exhibit highly anisotropic curvature ( 34 ) . In particular, local curvature magnitudes vary substantially across parameters (Figure 1 ), providing a proxy for local sensitivity. This observation suggests that an effective ZO optimization algorithm could leverage curvature information to prioritize informative parameter subsets and guide perturbations toward higher-curvature directions. However, achieving this remains challenging in the ZO setting, where each query yields only a scalar response, and neither gradients nor curvature can be directly accessed. In this work, we propose Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO) , a framework that tracks curvature signals online from scalar responses and leverages them to construct a sampling distribution for sparse perturbations. CurvZO reduces the variance of the sparse ZO gradient estimator by perturbing parameters with larger local curvature signals more frequently. Moreover, we develop an adaptive budget mechanism that dynamically adjusts the number of perturbed parameters according to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. The summary of our contributions is as follows: • Online Curvature Signal Tracking. We propose a curvature score to track curvature signals online from the scalar responses of ZO updates, which captures the local geometry of the loss landscape and provides guidance for selective perturbations in sparse ZO. • Curvature-Guided Sparse ZO Fine-Tuning. Leveraging online curvature scores, we design a curvature-guided sampling rule that assigns higher sampling probabilities to parameters with larger scores. This rule concentrates perturbations on higher-curvature directions, thereby reducing the variance of the sparse ZO gradient estimator. • Adaptive Budget Selection. We introduce an adaptive budget mechanism that adjusts the number of perturbed parameters based on the evolving curvature score distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. • Empirical Evaluation. We evaluate CurvZO on OPT and Llama models across diverse NLP tasks. CurvZO consistently outperforms ZO baselines and converges faster, improving accuracy by up to 4.4 points and reducing GPU hours by up to 2\times , while preserving ZO-level memory efficiency. 2 Preliminaries 2.1 Zeroth-Order Optimization ZO optimization has recently attracted significant attention in machine learning ( 27 ; 29 ; 9 ) . It estimates gradients using only a few forward passes, without requiring backpropagation. Crucially, ZO eliminates the need to store memory-intensive components required in FO training, such as intermediate activations, backward gradients, and optimizer states, thereby substantially reducing GPU memory consumption. A classical gradient estimator used in ZO methods is the Simultaneous Perturbation Stochastic Approximation (SPSA) ( 24 ) . Each ZO query yields a scalar response via finite-difference approximation: \Delta=\frac{\mathcal{L}(\boldsymbol{\omega}+\epsilon\boldsymbol{z};\mathcal{B})-\mathcal{L}(\boldsymbol{\omega}-\epsilon\boldsymbol{z};\mathcal{B})}{2\epsilon}, (1) where \boldsymbol{\omega}\in\mathbb{R}^{d} denotes the model parameters, and \mathcal{L}(\boldsymbol{\omega};\mathcal{B}) is the loss evaluated on a minibatch \mathcal{B} of size B , uniformly sampled from the training dataset \mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{|\mathcal{D}|} . Meanwhile, \boldsymbol{z}\in\mathbb{R}^{d} is a random perturbation sampled from \mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{d}) , and \epsilon is the perturbation scale. The ZO gradient estimator is given by \hat{\nabla}\mathcal{L}(\boldsymbol{\omega};\mathcal{B})=\Delta\boldsymbol{z}. (2) Then, this estimator can be plugged into standard FO optimizers. For example, ZO-SGD updates the parameters as \boldsymbol{\omega}^{t+1}=\boldsymbol{\omega}^{t}-\eta^{t}\hat{\nabla}\mathcal{L}(\boldsymbol{\omega}^{t};\mathcal{B}), (3) where \eta^{t}>0 is the learning rate at iteration t . 2.2 Sparsity in ZO Existing studies have established that ZO fine-tuning performance degrades as the dimensionality of the optimized parameters increases ( 17 ; 30 ) . To mitigate this,

Read the complete text