LoRDO : Distributed Low-Rank Optimization with Infrequent Communication
Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane · 2026
Read LoRDO : Distributed Low-Rank Optimization with Infrequent Communication by Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04396.txt.
Description
Distributed training chokes on synchronization, and Andrej Jovanović and 8 co-authors loosen the knot with LoRDO. Their 2026 framework merges low-rank optimization with infrequent communication. Workers updating locally, though, cannot compute the full-batch gradients that low-rank projections normally need.
Global projections from pseudo-gradients look superior but permanently cage the trajectory in a low-rank subspace. A full-rank quasi-hyperbolic update restores exploration outside that cage. The system reached near-parity with low-rank standard distributed training on language modeling at moderate scales, while cutting communication roughly tenfold. Bandwidth savings of that size reshape what training clusters can attempt.
Catalog record
| CREATOR | Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04396