Load Balancing for AI Training Workloads
Sarah McClure, Sylvia Ratnasamy, Scott Shenker · 2025
Read Load Balancing for AI Training Workloads by Sarah McClure, Sylvia Ratnasamy, Scott Shenker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21372.txt.
Introduction
Sarah McClure and co-authors compare traffic balancing for AI training in a 2025 paper. Training large models moves huge amounts of data between chips. The network that carries it must spread load evenly to stay fast. Many designs exist, but no clear winner was known.
The team tested leading designs apart from the systems around them. Splitting traffic packet by packet beat splitting by flow or burst. Host based and switch based spraying tied when nothing failed. When a link broke, the host based design recovered far better. None of the leaders reached the best possible queue scaling. The authors then present a rotation scheme that does reach it. Their switch version of it gained clear ground in tests.
Catalog record
| CREATOR | Sarah McClure, Sylvia Ratnasamy, Scott Shenker |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21372
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Load Balancing for AI Training Workloads Sarah McClure, Sylvia Ratnasamy, Scott Shenker Abstract We investigate the performance of various load balancing algorithms for large-scale AI training workloads that are running on dedicated infrastructure. The performance of load balancing depends on both the congestion control and loss recovery algorithms, so our evaluation also sheds light on the appropriate choices for those designs as well. 1 Introduction The near-universal adoption of TCP has long imposed an implicit requirement on networks: the stream of packets in a single TCP connection must be routed along a single network path . This requirement arises because splitting a connection across multiple paths could lead to a significant reordering of packets; in turn, a TCP sender might mistake the reordering for loss (because of the resulting duplicate ACKs) and then unnecessarily decrease the connection’s congestion window; finally, cutting the congestion window may do little to decrease the reordering, so the sender may repeatedly decrease the window resulting in a severe impact on the connection’s throughput [ 57 , 20 ] . Thus, in enterprise and WAN networks, sending a connection’s traffic along multiple paths was almost always avoided. Datacenter networks are different, in that they typically have many equal-cost paths. Sending packets along multiple equal-cost paths would not induce the level of reordering that occurs in more heterogeneous networks, so one might have expected such networks to depart from the dominant single-path paradigm. However, most datacenter networks have continued to follow this paradigm in spirit. ECMP [ 26 ] routes on five-tuples, which keeps a connection’s traffic on a single path, and more recent extensions like PLB [ 43 ] and flowlet switching [ 30 ] only change a connection’s route during idle periods so as to avoid reordering. More specifically, during nonidle periods flows are still constrained to a single path, but those paths can be changed ( e.g., to avoid congested links) during idle periods, thereby giving the load-balancing scheme a bit more flexibility without incurring reordering. This persistence of the single-path paradigm (with minor extensions) exists despite the fact that the single-path paradigms has serious drawbacks. In particular, requiring a TCP-connection (or a flowlet) to stay on a single path: (i) limits the capacity available to a connection, even if significant capacity exists on alternate paths [ 27 , 44 , 29 ] , (ii) can create hotspots that impact tail latency if several long-lived flows are sent along the same link [ 43 , 5 ] , and (iii) can lead to highly sub-optimal load balancing when flow sizes vary greatly (because some links are more heavily loaded than others) [ 5 , 22 ] . Not only are there serious drawbacks, but the community has developed viable technical alternatives to the single-path paradigm. As far back as the 1990s, researchers were already exploring modifications to TCP that successfully exploited multiple paths [ 27 , 44 ] and robustly tolerated packet reordering [ 57 ] . More recent research has explored transport designs for the datacenter that uses packet spraying [ 6 ] (in which a host or switch sends packets across all equal-cost paths; this is the antithesis to keeping connections on a single path) and packet trimming for loss detection [ 24 ] (in which a switch, when a packet arrives that would otherwise be dropped, strips the data from the packet and forwards the remaining header at high-priority). The reason that these alternative designs were not widely adopted is presumably because the traditional solution of ECMP, particularly when combined with approaches like PLB, provided “good enough” performance for the then-current workloads. Thus, there was little need to depart from the single-path paradigm, even if this paradigm was not optimal. However, the advent of AI training workloads has greatly changed the performance requirements on AI-oriented datacenter networks, which in turn has opened the door to innovations that violate the single-path paradigm. In what follows, when we refer to AI workloads we mean large-scale AI training workloads that are run on their own dedicated infrastructure. While we discuss AI workloads at greater length in § 2 , here we merely note that these workloads have two characteristics that accentuate the limitations of the single-path paradigm. First, the vast bulk of the AI traffic is in long-lived flows that do not have idle periods, so techniques like PLB do not ameliorate the rigidity of the single-path paradigm ( i.e., as mentioned above, if two long-lived flows are hashed by ECMP to the same link, PLB cannot reroute them because there are no idle periods, so tail-latencies can be strongly affected). Second, the performance requirements of AI workloads are extreme, requiring the full completion of all flows in a communication collective [ 12 , 21 , 3 ] (which we define in § 2 ). Thus, the techniques that mitigated some of ECMP’s problems ( e.g., PLB) are no longer effective , and the performance degradation caused by strictly adhering to limiting flows (or flowlets) to a single path is no longer acceptable . This need for better performance for AI workloads has led to active exploration, mostly in industry, of designs that go beyond the single-path paradigm. Here we mention two efforts in particular. The Ultra Ethernet Consortium (UEC) [ 51 ] is considering new transport designs that include packet spraying and packet trimming [ 2 , 3 ] . Nvidia has developed SpectrumX which uses a switch-based packet-spraying approach. These efforts suggest that we are finally seeing a broad appetite in industry to consider load balancing techniques that no longer restrict packet flows to a single path. When designing load-balancing schemes for AI workloads, the goal is to minimize collective completion time (CCT) for AI traffic. Note that this performance goal is quite different from the goals for traditional datacenter workloads where the focus is on: (i) per-flow measures such as flow-completion time and per-flow fairness (as compared to AI’s focus on collective completion time) and (ii) maximizing utilization so that the network is being used efficiently (utilization is less relevant for AI workloads because the network is a small component of the overall cost of an AI cluster, so the network need not be cheap and fully utilized, it only needs to complete collectives quickly). While progress has been made in developing specific solutions to meet this goal, such as the two previously mentioned efforts (UEC and SpectrumX), we were surprised to find that many basic design questions remain open. For instance, while it seems clear that packet spraying is a robust technique for spreading load along all suitable paths, we were not able, in our literature search, to find answers to such key questions as: (i) How much better is packet spraying versus the state-of-the-art ECMP+PLB designs? (ii) Do the benefits of (various forms of) packet spraying remain when we introduce failures or the workload creates incast? (iii) Is it better to implement packet spraying in switches (as in SpectrumX and pFabric [ 6 ] ) or in hosts (as in NDP)? (iv) Do we need packet trimming for loss detection or are more incremental changes to TCP-like protocols adequate? (v) And what forms of congestion control are appropriate in