CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training
Tiancheng Chen, Ales Kubicek, Langwen Huang, Torsten Hoefler · 2025
Read CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training by Tiancheng Chen, Ales Kubicek, Langwen Huang, Torsten Hoefler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00217.txt.
Description
Chen, Kubicek, Huang, and Hoefler tackle scheduling when training a large language model must spill across several datacenters, in a 2025 paper. Distance adds delay and throttles bandwidth, and standard pipeline schedules waste the hardware. Their framework, CrossPipe, models those network costs directly and searches for schedules that overlap data-parallel traffic with pipeline stages.
It offers both an exact solver and a fast greedy method, run on an engine that separates scheduling logic from communication details. Training time fell by up to a third against traditional schedules under the same memory limits. With more memory, performance approached delay-free ideals despite the distance.
Catalog record
| CREATOR | Tiancheng Chen, Ales Kubicek, Langwen Huang, Torsten Hoefler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00217