Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale

Sihwan Park, Jihun Yun, Sungyub Kim, Souvik Kundu, Eunho Yang · 2025

Read Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale by Sihwan Park, Jihun Yun, Sungyub Kim, Souvik Kundu, Eunho Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19099.txt.

Introduction

Sihwan Park and four co-authors study a training method that works without direct slope measures, in a 2025 paper. Such methods suit black-box problems and tuning of large language models. Their weakness is slow progress, since the slope estimates wobble a lot. Limiting changes to a small subspace can help, but the reason was unclear.

The team built a single theory frame for these methods. High size of the model, they show, is the main bottleneck. A new idea, subspace alignment, explains how limits cut noise and speed up training. Most limit types share the same speed in theory. So the authors stress practical design, and they propose a method that updates one block of weights at a time. Tests sped up tuning of a large model by a wide margin.

Catalog record

CREATORSihwan Park, Jihun Yun, Sungyub Kim, Souvik Kundu, Eunho Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE155 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19099

Browse the full library · About the Tower of Babel