Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning
Qitao Tan, Jun Liu, Zheng Zhan, Caiwen Ding, Yanzhi Wang, Xiaolong Ma, Jaewoo Lee, Jin Lu, Geng Yuan · 2025
Read Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning by Qitao Tan, Jun Liu, Zheng Zhan, Caiwen Ding, Yanzhi Wang, Xiaolong Ma, Jaewoo Lee, Jin Lu, Geng Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03304.txt.
Description
Qitao Tan and co-authors speed up a memory-light way of fine-tuning large language models, in a 2025 paper. Standard fine-tuning stores backward-pass data and eats memory, which limits use on modest hardware. Zeroth-order methods skip the backward pass and guess slopes from forward runs alone, saving memory but trailing in speed and accuracy.
Comparing the two methods layer by layer, the team found their updates diverge in a patterned way. Their method, DiZO, reshapes the zeroth-order updates per layer so their sizes match what each layer needs. Training hours fell by up to about half across data sets, and tuned models matched or beat strong baselines, at times even passing the memory-heavy standard method.
Catalog record
| CREATOR | Qitao Tan, Jun Liu, Zheng Zhan, Caiwen Ding, Yanzhi Wang, Xiaolong Ma, Jaewoo Lee, Jin Lu, Geng Yuan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-03304