Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale
Sihwan Park, Jihun Yun, Sungyub Kim, Souvik Kundu, Eunho Yang · 2025
Read Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale by Sihwan Park, Jihun Yun, Sungyub Kim, Souvik Kundu, Eunho Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19099.txt.
Introduction
Sihwan Park and four co-authors study a training method that works without direct slope measures, in a 2025 paper. Such methods suit black-box problems and tuning of large language models. Their weakness is slow progress, since the slope estimates wobble a lot. Limiting changes to a small subspace can help, but the reason was unclear.
The team built a single theory frame for these methods. High size of the model, they show, is the main bottleneck. A new idea, subspace alignment, explains how limits cut noise and speed up training. Most limit types share the same speed in theory. So the authors stress practical design, and they propose a method that updates one block of weights at a time. Tests sped up tuning of a large model by a wide margin.
Catalog record
| CREATOR | Sihwan Park, Jihun Yun, Sungyub Kim, Souvik Kundu, Eunho Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 155 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19099