Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc’Aurelio Ranzato, Paul Barham · 2025
Read Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch by Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc’Aurelio Ranzato, Paul Barham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18512.txt.
Description
Arthur Douillard and a large team of engineers improve a method for training big language models in this two thousand twenty-five paper. Training is usually spread over many chips in one place, linked by fast cables. A newer scheme, DiLoCo, lets groups of chips work apart and sync only now and then. That frees training to use slower links between sites. But the sync steps still cause stalls.
The team presents a streaming version that overlaps communication with computing. Workers send their updates while training goes on, rather than pausing. This hides much of the waiting time. In effect, distributed training gains speed almost for free. Their tests target large language model training across worker groups. The work matters as models grow and training spreads across data centers.
Catalog record
| CREATOR | Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc’Aurelio Ranzato, Paul Barham |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18512