Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

Sijia Liu, Yicheng Lang, Soumyadeep Pal, Changsheng Wang, Yancheng Huang, Chongyu Fan, James Diffenderfer, Bhavya Kailkhura, Yihua Zhang · 2026

Read Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered by Sijia Liu, Yicheng Lang, Soumyadeep Pal, Changsheng Wang, Yancheng Huang, Chongyu Fan, James Diffenderfer, Bhavya Kailkhura, Yihua Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15622.txt.

Introduction

Sijia Liu and eight colleagues say zeroth-order optimization is underexplored, not underpowered. It learns from function values with no backpropagation. The team blames narrow, estimator-first designs for its poor name. They set out six positions across algorithms, systems, and evaluation. Openings include subspace and spectral views for clear variance cuts with gentle query growth. The forward-only build suits communication-light, pipeline-friendly, resource-tight training. Cleaner tests split from task complexity round out the case. The paper urges a rethink around these strengths.

Catalog record

CREATORSijia Liu, Yicheng Lang, Soumyadeep Pal, Changsheng Wang, Yancheng Huang, Chongyu Fan, James Diffenderfer, Bhavya Kailkhura, Yihua Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50012
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15622

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered Sijia Liu, Yicheng Lang, Soumyadeep Pal, Changsheng Wang, Yancheng Huang, Chongyu Fan, James Diffenderfer, Bhavya Kailkhura, Yihua Zhang Abstract Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered . We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance–query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization. † † affiliationnotice: Equal contribution 1 Introduction Zeroth-order ( ZO ) optimization refers to the gradient-free counterpart of first-order ( FO ) optimization. That is, instead of accessing exact or stochastic FO gradients, ZO methods approximate descent directions using function-value–based gradient estimators ( 80 ; 18 ; 62 ; 22 ; 54 ) . Although this function access may appear restrictive, ZO optimization is broadly relevant to deep learning for two reasons. First, ZO optimization offers a principled alternative to backpropagation (BP) for efficient learning ( 59 ; 102 ; 84 ) . By eliminating backward passes, ZO methods can substantially reduce activation memory, simplify execution pipelines, and enable training under hardware, system, or deployment constraints where BP is impractical or inefficient ( 31 ; 109 ) . Second, ZO optimization is indispensable for black-box and gray-box learning ( 103 ; 8 ; 58 ; 35 ; 83 ) , where FO gradients are unavailable, unreliable, or prohibitively expensive. This setting commonly arises in learning scenarios involving black-box components, such as simulators or ML-as-a-service (MLaaS) systems. Prior to 2023, ZO optimization established its empirical credibility in deep learning through input-level black-box applications , most notably adversarial example generation ( 9 ; 35 ) and input prompting ( 83 ; 88 ) under black-box models or query-only APIs. These successes positioned ZO optimization as a practical and competitive tool when gradient information is inaccessible. However, such applications typically operate in relatively low-dimensional input spaces ( e.g. , images or token sequences), where the core scalability challenges of ZO optimization, such as gradient estimation variance and query cost, are largely suppressed. Figure 1 : Schematic overview of this position paper. (Left) Publication trends (papers per year) for works with “ZO optimization” in the title in arXiv cs.AI and cs.LG (machine learning). (Right) Conceptual organization of our positioning points (P1–P6). A qualitative shift occurred with the emergence of memory-efficient fine-tuning via ZO optimization ( 59 ) , which moved ZO methods from input-space manipulation to the much higher-dimensional regime of weight-level model training . This shift has sparked a surge of ZO research since 2023, as shown in Fig. 1 (Left) . By replacing BP with forward-only passes, ZO optimization can fine-tune large pretrained models, while significantly reducing memory overhead ( 102 ) . Thus, a growing literature now aims to scale ZO optimization for large-model fine-tuning, proposing increasingly sophisticated ZO gradient estimators and variance-reduction techniques ( 56 ; 96 ; 104 ; 11 ) . Beyond memory efficiency, another key advantage of ZO training is flexibility : it enables end-to-end optimization of hybrid systems composed of both ML and non-ML components, where differentiability is neither natural nor desirable. This capability is particularly relevant to emerging paradigms such as physics-informed neural networks ( 68 ) , digital twins ( 60 ; 58 ) , and agentic AI ( 3 ) , whose pipelines often combine symbolic logic, simulators, tools, or environment interactions with neural modules. However, progress along this direction remains limited ( 108 ; 8 ) , largely because these use cases often require training from scratch with ZO optimization. As a result, existing efforts are mostly confined to small architectures ( e.g. , CNNs) rather than large-scale models ( 8 ; 97 ) . This has also fueled skepticism in the community and raises a natural question: Is ZO optimization fundamentally limited in deep learning, or have we simply not explored it in the right way? In this work, we take the position that ZO optimization is underexplored, not underpowered. We argue that many perceived limitations of ZO optimization stem not from intrinsic weaknesses of gradient-free learning, but from myopic development practices, most notably, an overemphasis on gradient-estimator–centric design in the element-wise parameter space. Existing approaches overlook broader algorithmic, system-level, and evaluation opportunities that are critical to advancing ZO optimization in deep learning. In this work, we articulate six positioning arguments (P1–P6) spanning the algorithm–system–evaluation stack; see a schematic overview in Fig. 1 (Right) . Specifically, positions P1–P3 characterize the feasibility boundaries of current ZO optimization (see Sec. 2 ). From an algorithmic perspective, P1 and P2 are proposed to pinpoint the most critical design factor in existing studies–variance control in ZO gradient estimation. From an evaluation perspective, P3 highlights an often-overlooked feasibility diagnostic: the directional-derivative–based formulation can serve as a principled reference for assessing the difficulty of ZO optimization. Positions P4–P6 call for rethinking and advancing ZO optimization from complementary perspectives (see Sec. 3 ). From an algorithmic perspective, P4 challenges the prevailing practice of operating ZO methods in the original parameter space, which overlooks the substantial scalability advantages of subspace learning. From an infrastructure perspective, P5 advocates exploiting the simplicity of ZO forward-only execution as a systems advantage, including natural parallelism and communication efficiency, properties that are highly aligned with distributed and resource-constrained learning environments. From an evaluation perspective, P6 calls for disentangling the performance of ZO optimization from its associated task complexity to avoid obscured assessments of ZO’s true optimization power. Furthermore, we distill P1–P6 into a concise set of actionable calls in Sec. 4 and contrast our position with several alternative viewpoints in Sec. 5 . Conflict of Interest Disclosure The authors declare that they have no conflicts of interest. 2 Foundations and Feasibility Boundaries of ZO Gradient Estimator-Centric Design Most existing ZO methods rely on query-based gradient estimators that approximate FO gradients using only function evaluations, enabling FO optimization frameworks to be reused in the ZO setting. In this section, we briefly review the foundations of this estimator-centric paradigm and argue that, even within it, several critical considerations have been overlooked and deserve closer attention. Consider the minimization problem \min_{\mathbf{x}\in\mathbb{R}^{d}}f(\mathbf{x}) , where f(\mathbf{x}) is an objective function defined over a d -dimensional optimization variable \mathbf{x}\in\mathbb{R}^{d} . A vanilla ZO gradient estimator for approximating the FO gradient \nabla_{\mathbf{x}}f(\mathbf{x}) , known as a randomized gradient estimator (RGE) , is given by \displaystyle\hat{\nabla}_{\mathbf{x}}f(\mathbf{x})=\frac{f(\mathbf{x}+\mu\mathbf{u})-f(\mathbf{x})}{\mu}\,\mathbf{u}, (RGE) where \hat{\nabla}_{\mathbf{x}}f(\mathbf{x})

Read the complete text