Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective

Zhe Li, Bicheng Ying, Zidong Liu, Haibo Yang · 2026

Read Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective by Zhe Li, Bicheng Ying, Zidong Liu, Haibo Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03373.txt.

Description

Zhe Li, Bicheng Ying, Zidong Liu, and Haibo Yang explain why zeroth-order optimization, long thought to slow badly with model size, can fine-tune billion-parameter language models. Classic theory predicts a dimension penalty versus first-order methods, yet practice keeps succeeding at huge scale.

Deriving one-step learning dynamics, they show the empirical neural tangent kernel governs progress, and its entries are tangent vectors projected onto a random low-dimensional subspace. By a classic embedding lemma, fidelity depends on the number of perturbations and the output size, not the parameter count. That dimension-free property resolves the paradox in theory.

Catalog record

CREATORZhe Li, Bicheng Ying, Zidong Liu, Haibo Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03373

Browse the full library · About the Tower of Babel