Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory
Yao Shu, Jian Mu, Zhongxiang Dai · 2026
Read Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory by Yao Shu, Jian Mu, Zhongxiang Dai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10658.txt.
Description
Yao Shu, Jian Mu, and Zhongxiang Dai give a random gradient-shaping theory explaining why zeroth-order tuning may lose less than first-order descent in ongoing learning. Their analysis shows finite differences expose a raw shape mean-aligned with first-order gradients, while a norm-matched benchmark fixes the mean squared tuning norm. Loss is thus controlled by how the tuning shape exposes memory bend.
For norm-matched zeroth-order updates, the mean shaped memory bend obeys an exact rule preserving the even memory floor while contracting only the uneven component. Projecting this rule onto the incoming gradient yields the seen loss gap. Zeroth-order improves mean loss exactly when the first-order direction has above-average memory bend, by a query-dependent fraction of that excess. The mechanism also transfers in code: RISE applies the calibrated zeroth-order shape to exact first-order gradients inside parameter blocks.
Catalog record
| CREATOR | Yao Shu, Jian Mu, Zhongxiang Dai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108312 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10658