CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training

Tiancheng Chen, Ales Kubicek, Langwen Huang, Torsten Hoefler · 2025

Read CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training by Tiancheng Chen, Ales Kubicek, Langwen Huang, Torsten Hoefler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00217.txt.

Description

Chen, Kubicek, Huang, and Hoefler tackle scheduling when training a large language model must spill across several datacenters, in a 2025 paper. Distance adds delay and throttles bandwidth, and standard pipeline schedules waste the hardware. Their framework, CrossPipe, models those network costs directly and searches for schedules that overlap data-parallel traffic with pipeline stages.

It offers both an exact solver and a fast greedy method, run on an engine that separates scheduling logic from communication details. Training time fell by up to a third against traditional schedules under the same memory limits. With more memory, performance approached delay-free ideals despite the distance.

Catalog record

CREATORTiancheng Chen, Ales Kubicek, Langwen Huang, Torsten Hoefler
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00217

Browse the full library · About the Tower of Babel