Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc’Aurelio Ranzato, Paul Barham · 2025

Read Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch by Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc’Aurelio Ranzato, Paul Barham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18512.txt.

Description

Arthur Douillard and a large team of engineers improve a method for training big language models in this two thousand twenty-five paper. Training is usually spread over many chips in one place, linked by fast cables. A newer scheme, DiLoCo, lets groups of chips work apart and sync only now and then. That frees training to use slower links between sites. But the sync steps still cause stalls.

The team presents a streaming version that overlaps communication with computing. Workers send their updates while training goes on, rather than pausing. This hides much of the waiting time. In effect, distributed training gains speed almost for free. Their tests target large language model training across worker groups. The work matters as models grow and training spreads across data centers.

Catalog record

CREATORArthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc’Aurelio Ranzato, Paul Barham
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18512

Browse the full library · About the Tower of Babel