ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
Chuiyang Meng 1 , Ming Tang 2 , and Vincent W.S. Wong 1 · 2026
Read ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation by Chuiyang Meng 1 , Ming Tang 2 , and Vincent W.S. Wong 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04436.txt.
Description
Meng and colleagues federate language model tuning with ZorBA, a zeroth order method. Local updates burden client video memory, and frequent exchange burdens communication. Zeroth order passes drop gradient storage at clients. Different block subsets per client speed convergence and cut memory, while shared seeds and finite differences cut communication. Theory links block choices to speed and memory, and tests show VRAM use falls by up to 62.41 percent against three baselines.
Catalog record
| CREATOR | Chuiyang Meng 1 , Ming Tang 2 , and Vincent W.S. Wong 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90012 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04436