Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters
Foteini Strati, Zhendong Zhang, George Manos, Ixeia Sánchez Périz, Qinghao Hu, Tiancheng Chen, Berk Buzcu, Song Han, Pamela Delgado, Ana Klimovic · 2025
Read Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters by Foteini Strati, Zhendong Zhang, George Manos, Ixeia Sánchez Périz, Qinghao Hu, Tiancheng Chen, Berk Buzcu, Song Han, Pamela Delgado, Ana Klimovic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17096.txt.
Introduction
Foteini Strati and co-authors present Sailor, a system that automates model training across mixed clusters in two thousand twenty-five. Demand for top graphics chips far outstrips supply in any one data center zone. Chips of mixed grades, spread across zones, could fill the gap at fair cost, but they train unevenly.
Slow stragglers and a huge space of setup choices make such training hard to run by hand. Sailor searches that space with a fast planner, predicts run time and memory use by simulation, and runs the job on a frame that handles each kind of mismatch. The aim is high throughput at a sane cost on whatever chips are free, even as the pool changes mid-run.
Catalog record
| CREATOR | Foteini Strati, Zhendong Zhang, George Manos, Ixeia Sánchez Périz, Qinghao Hu, Tiancheng Chen, Berk Buzcu, Song Han, Pamela Delgado, Ana Klimovic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17096
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters Foteini Strati, Zhendong Zhang, George Manos, Ixeia Sánchez Périz, Qinghao Hu, Tiancheng Chen, Berk Buzcu, Song Han, Pamela Delgado, Ana Klimovic Abstract . The high GPU demand of ML training makes it hard to allocate large homogeneous clusters of high-end GPUs in a single availability zone. Leveraging heterogeneous GPUs available within and across zones can improve throughput at a reasonable cost. However, training ML models on heterogeneous resources introduces significant challenges, such as stragglers and a large search space of possible job configurations. Current systems lack support for efficiently training models on heterogeneous resources. We present Sailor, a system that automates distributed training over heterogeneous, geo-distributed, and dynamically available resources. Sailor combines an efficient search space exploration algorithm, accurate runtime and memory footprint simulation, and a distributed training framework that supports different types of heterogeneity to optimize training throughput and cost. Conference: ACM SIGOPS 31st Symposium on Operating Systems Principles; October 13–16, 2025; Seoul, Republic of Korea ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP ’25), October 13–16, 2025, Seoul, Republic of Korea DOI: 10.1145/3731569.3764839 ISBN: 979-8-4007-1870-0/2025/10 CCS: Computing methodologies Machine learning † † cc-license: by 1. Introduction GPUs are in high demand for large-scale Machine Learning (ML). As ML models continue to grow exponentially in size, they require an increasing number of GPUs to train and fine-tune. This high demand makes it difficult for model developers to allocate the desired number of accelerators to train models at high throughput in public clouds or enterprise clusters ( Um et al., 2024 ; Strati et al., 2024a ; Yang et al., 2023 ) . Datacenters typically host a variety of GPU types and generations, spread across geographic regions ( Cloud, 2024 ; Jia et al., 2022 ; Weng et al., 2022 ; Jeon et al., 2019 ; Wu et al., 2024 ) . Yet model developers tend to restrict model training to homogeneous clusters of GPUs, since state-of-the-art distributed training frameworks like Megatron-LM ( Shoeybi et al., 2020 ) and DeepSpeed ( deepspeedai, 2025 ) assume homogeneous GPUs and inter-node bandwidth. The demand for large, homogeneous GPU clusters compounds the scarcity of high-end GPUs. Figure 1 . When homogeneous resources are limited, using heterogeneous GPUs (A100, V100) or multiple zones can increase OPT-350M training throughput (c3-c4) at low cost (black line). However, when the resource topology and job parallelization are not well selected, iteration time and monetary cost may increase significantly (c5-c6). Allowing a training job to run on heterogeneous GPU types and/or GPUs distributed across zones (i.e., with heterogeneous inter-node bandwidth) can give model developers access to more GPUs per job to increase training throughput. For example, consider a model developer who seeks to maximize training throughput by using 32 A100 GPUs ( c2 in Figure 1 ), but discovers that only 16 A100s ( c0 in Figure 1 ) are currently available in one zone. Using heterogeneous GPU generations (e.g., c3 uses an additional 16 V100s in the zone) or multi-zone configurations (e.g., c4 uses 32 A100s spread across 2 zones within a region) increases throughput by 1.15 \times and 1.87 \times , respectively, with a moderate increase in monetary cost per iteration (black line). However, supporting heterogeneous, geo-distributed resources introduces several challenges. First, heterogeneous GPU types and placements across zones exponentially expand the configuration search space. Searching for an optimal configuration requires jointly optimizing the resource allocation and job parallelization plan (e.g., data/pipeline/tensor parallelism degrees). For example, in Figure 1 , although c5 uses the same number of GPUs as c3 , it achieves much lower throughput due to a suboptimal parallelization plan. The search must also consider the cost caused by extra resources and data transfers. Cloud providers charge significant fees for inter-zone and inter-region data transfers ( AWS, 2025 ; Cloud, 2025 ; Azure, 2025 ) , which impacts geo-distributed configurations. In Figure 1 , c6 uses the same number of GPUs and parallelization strategy as c4 , but spreads training across regions (instead of zones within the same region), which increases cost. Furthermore, as resource availability changes dynamically in datacenters, due to variable demand and node failures or preemptions, it is necessary to navigate this vast search space quickly ( Thorpe et al., 2023 ; Wagenländer et al., 2024 ) . Figure 2 shows the varying number of A100 GPUs that we were able to allocate over an 8-hour period, out of the 8 A100s that we continuously requested in 2 different zones in Google Cloud. Such changes in resource availability require frequently re-evaluating the job configuration search space. Thus, model developers require a system to quickly navigate the large search space of heterogeneous (and homogeneous) configurations with dynamic resource availability, optimizing for the user’s performance-cost objective, while satisfying constraints (e.g., budget limits). Second, profiling many candidate configurations to evaluate their throughput is prohibitively expensive and time-consuming. Hence, it is critical to accurately estimate the iteration time of a candidate configuration. This is challenging in heterogeneous environments, as differences in peak FLOPS, memory capacity, CPU-GPU interconnects, number of GPUs per node, and inter-node bandwidth typically lead to stragglers , which can significantly limit throughput. More importantly, variability in memory capacity per GPU may cause out-of-memory (OOM) errors in some GPUs, disrupting the entire training job. Simulating iteration time and checking that job configurations are valid (i.e., will not cause OOM) requires correctly modeling stragglers and per-GPU memory footprints. Finally, after finding an appropriate resource allocation and parallelization plan for a training job given the available resources, model developers need to be able to run this job configuration in a distributed training framework (such as Megatron ( Shoeybi et al., 2020 ) ). We find that optimal configurations for jobs running on heterogeneous resource topologies often include heterogeneous parallelism degrees per stage to load-balance the compute and memory capacity of different GPU types. Today’s state-of-the-art distributed training frameworks need to be adapted to support such heterogeneous job configurations. Furthermore, as resource availability can change frequently (e.g., when using spot instances ( Thorpe et al., 2023 ) ), the training framework must be able to quickly reconfigure jobs. Existing systems do not adequately solve these challenges. First, current works do not co-optimize the resource allocation with the job parallelization plan. Instead, systems like Aceso ( Liu et al., 2024 ) , Galvatron ( Miao et al., 2022 ) , and others in Table 1 expect the user to select a fixed resource allocation for which the system recommends a job parallelization plan. Most systems also do not consider heterogeneous resource topologies. Recent systems like Atlas ( Palak et al., 2024a ) , DTFM ( Yuan et al., 2023 ) , Metis ( Um et al., 2024 ) , and FlashFlex ( Yan et al., 2024 ) optimize parallelization for heterogeneous GPUs or geo-distributed setups, but they suffer from prohibitively long search times (up to hours for configurations with 10s of GPUs) ( Um et al., 2024 ) , or suboptimal cost functions ( Yan et al., 2024 ; Yuan et al., 2023 ) , making them unsuitable for environments with dynamic resource availability. Second, existing systems rely on inaccurate simulators to estimate the training throughput and