Symphony : Taming Step Misalignments in the Network for Ring-based Collective Operations

Yuze Jin, Xin Zhe Khooi, Ruyi Yao, Mun Choon Chan · 2026

Read Symphony : Taming Step Misalignments in the Network for Ring-based Collective Operations by Yuze Jin, Xin Zhe Khooi, Ruyi Yao, Mun Choon Chan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16880.txt.

Introduction

Jin and three colleagues address a timing flaw in ring-based collective operations, the workhorse pattern for distributed AI training. Network jitter leaves some steps ahead of others and drags out completion. Their in-network Symphony design tracks each job's pipeline progress and throttles racing flows so laggards catch up, cutting communication time by up to 54 percent in simulation.

Catalog record

CREATORYuze Jin, Xin Zhe Khooi, Ruyi Yao, Mun Choon Chan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59544
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16880

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Symphony : Taming Step Misalignments in the Network for Ring-based Collective Operations Yuze Jin, Xin Zhe Khooi, Ruyi Yao, Mun Choon Chan Abstract . Ring-based collective operations are widely used in distributed AI training due to their efficient bandwidth utilization. While ring communication excels at pipelining, its performance is heavily dependent on having synchronized step-wise progression. This presents a mismatch to the underlying network conditions in practice: collective operations are vulnerable to network jitter and congestion, leading to step misalignment and increased collective completion time. To that end, we propose Symphony , an in-network solution that detects pipeline step misalignment and mitigates its impact. Symphony introduces (1) a lightweight mechanism to track per-job pipeline progress and (2) a novel use of congestion signals to selectively throttle outpacing flows, allowing lagging flows to catch up without global coordination. Through simulations using Astra-Sim, we show that Symphony effectively mitigates step misalignments in ring-based collectives, resulting in up to 54% improvement in job/collective communication time. Finally, we prototype and validate Symphony on an Intel Tofino2 programmable switch to demonstrate its practicality. 1. Introduction Figure 1 . Ideal vs. Realistic behavior of ring - based collective operations. (a–b) Even small path imbalances (e.g., ECMP hash collisions) cause uneven link sharing and flow accumulation, unlike the ideal case. (c) Logical view of a single ring. (d) Under ideal conditions, steps progress in perfect lockstep. (e) In practice, congestion on a single link delays a step, causing subsequent steps to overlap and creating pipeline bubbles that amplify misalignment. Training modern AI models, such as Large Language Models (LLMs), is resource-intensive and often spans weeks to months, requiring tens of thousands of GPUs running in parallel ( Si et al. 2026 ; Jiang et al. 2024 ) . At this scale, training becomes communication-heavy: gradients and parameters must be exchanged across GPUs every iteration. In production clusters, communication bottlenecks often consume a significant portion of the total training iteration time, ranging from 16% to over 50% depending on the model size and parallelism strategy ( Aaron et al. 2024 ; Wu et al. 2024 ; Qian et al. 2024 ) . To fully utilize available bandwidth ( Huang et al. 2024 ; Zong et al. 2025 ) , today’s large-scale distributed AI training and inference workloads ( NVIDIA 2025 ; Nvidia 2022 ; Romero et al. 2022 ; Sergeev and Balso 2018 ; Baidu Research 2017 ; Gibiansky 2017 ) rely heavily on ring-based collective operations such as Ring-AllReduce. This is because ring-based collective operations excel at pipelining data, where different parts of a message (chunks) are sent concurrently across different links. Through pipelining, not only can the available network bandwidth be fully utilized, but the latency of large messages can also be effectively amortized. For ring-based collective operations to achieve their theoretical performance, they rely on the tight synchronization of the pipeline steps: concurrent transfers between adjacent nodes must proceed in lockstep . Therefore, even minor network perturbations, such as ECMP hash collisions, transient congestion, physical-layer packet losses ( Joshi et al. 2023 ; Lin et al. 2024 ; Li et al. 2025 ) , switch/link failures ( Jiang et al. 2024 ; Aaron et al. 2024 ) , or contention from background traffic in multi-tenant clusters ( Wu et al. 2024 ) , can cause step misalignment , where flows from different steps overlap on the same bottleneck link. This overlap causes bandwidth contention, triggers cascading delays, and dramatically inflates collective completion time (CCT). Unfortunately, today’s data center networks provide limited support for the synchronization needs of ring-based collectives. Existing mechanisms optimize for aggregate throughput and link utilization through techniques like perfecting load balancing ( Song et al. 2023 ; Ghorbani et al. 2017 ; Alizadeh et al. 2014 ) and congestion control ( Li et al. 2019 ; Zhu et al. 2015 ) , to squeeze every bit of utilization out of the networking fabric. However, such approaches blindly accelerate all flows, including those already ahead, and thus widen progress gaps and exacerbate misalignment. Host - side schedulers ( Shah et al. 2023 ; Cao et al. 2024 ; Wu et al. 2024 ) and application - level straggler mitigation techniques ( Harlap et al. 2016 ; Tandon et al. 2017 ) also fall short because they lose visibility and control once packets enter the network fabric ( Huang et al. 2017 ; Gangidi et al. 2024 ) . In fact, stragglers remain an important issue for LLM training ( Jiang et al. 2024 ; Qian et al. 2024 ; Warraich et al. 2025 ) . Key insight: Improving the performance of ring-based collective operations is not just about pushing more bandwidth; it requires the explicit control of the timing of flows so that steps stay synchronized as much as possible. To this end, we propose Symphony , an in-network mechanism that detects and mitigates step misalignment for ring-based collective operations. Symphony introduces a novel control strategy to restore pipeline alignment, where we throttle outpacing flows to dynamically free up bandwidth for lagging flows (or “stragglers”) to catch up. To achieve this, Symphony introduces two key ideas: (1) progress tracking : a lightweight per-job mechanism that infers real-time step progress using only packet metadata, and (2) selective throttling : a targeted use of congestion signaling (i.e., ECN marking) to throttle outpacing flows, allowing lagging flows to catch up without explicit coordination. Through extensive evaluations using Astra-Sim ( Won et al. 2023 ; Rashidi et al. 2020 ) with NS-3 ( nsnam 2025 ) , we show that Symphony effectively minimizes step alignments across various workloads, resulting in improved job and collective completion times (JCT/CCT) of up to 54%. To demonstrate Symphony ’s practical deployability, we prototype Symphony on an Intel Tofino2 ( Intel Corporation n d ) programmable switch, and show that Symphony can track step progress efficiently and throttle outpacing flows effectively to allow lagging flows to catch up. Contributions. This paper is the first to identify and quantify step misalignment as a hidden bottleneck in ring - based collective operations. Building on this insight, we introduce Symphony , a practical in - network mechanism, validated with a Tofino2 hardware prototype, that detects and mitigates step misalignments through a novel use of ECN marking to realize the selective throttling of outpacing flows. Figure 2 . Degree of step overlap and normalized step completion rates under different network conditions. 2. Background and Motivation 2.1. A primer on ring-based primitives Ring-based algorithms are commonly adopted for bandwidth-intensive LLM workloads due to their optimal link utilization via concurrent transmission ( NVIDIA 2025 ; Nvidia 2022 ; Hu et al. 2025 ) . Taking Ring All-Reduce as an example, each of the N nodes divides their data into N equal-sized chunks and exchanges data chunks over 2(N-1) pipelined steps. In each step, every node simultaneously transmits to its successor and receives from its predecessor, optionally undergoing computation (e.g., reduction) at each hop, as shown in fig. 1 c. Crucially, this throughput optimality relies on strict alignment: bandwidth is fully utilized only when concurrent flows start and finish in lockstep, leaving no links idle ( fig. 1 d). In practice, modern AI clusters deploy high-performance hosts equipped with

Read the complete text