LoRDO : Distributed Low-Rank Optimization with Infrequent Communication

Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane · 2026

Read LoRDO : Distributed Low-Rank Optimization with Infrequent Communication by Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04396.txt.

Description

Distributed training chokes on synchronization, and Andrej Jovanović and 8 co-authors loosen the knot with LoRDO. Their 2026 framework merges low-rank optimization with infrequent communication. Workers updating locally, though, cannot compute the full-batch gradients that low-rank projections normally need.

Global projections from pseudo-gradients look superior but permanently cage the trajectory in a low-rank subspace. A full-rank quasi-hyperbolic update restores exploration outside that cage. The system reached near-parity with low-rank standard distributed training on language modeling at moderate scales, while cutting communication roughly tenfold. Bandwidth savings of that size reshape what training clusters can attempt.

Catalog record

CREATORAndrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04396

Browse the full library · About the Tower of Babel