Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective
Zhe Li, Bicheng Ying, Zidong Liu, Haibo Yang · 2026
Read Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective by Zhe Li, Bicheng Ying, Zidong Liu, Haibo Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03373.txt.
Description
Zhe Li, Bicheng Ying, Zidong Liu, and Haibo Yang explain why zeroth-order optimization, long thought to slow badly with model size, can fine-tune billion-parameter language models. Classic theory predicts a dimension penalty versus first-order methods, yet practice keeps succeeding at huge scale.
Deriving one-step learning dynamics, they show the empirical neural tangent kernel governs progress, and its entries are tangent vectors projected onto a random low-dimensional subspace. By a classic embedding lemma, fidelity depends on the number of perturbations and the output size, not the parameter count. That dimension-free property resolves the paradox in theory.
Catalog record
| CREATOR | Zhe Li, Bicheng Ying, Zidong Liu, Haibo Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.03373