Improving Injection-Throttling Mechanisms for Congestion Control for Data-center and Supercomputer Interconnects
Cristina Olmedilla, Jesus Escudero-Sahuquillo, Pedro J. Garcia, Francisco J. Quiles, Jose Duato · 2025
Read Improving Injection-Throttling Mechanisms for Congestion Control for Data-center and Supercomputer Interconnects by Cristina Olmedilla, Jesus Escudero-Sahuquillo, Pedro J. Garcia, Francisco J. Quiles, Jose Duato (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05149.txt.
Introduction
Cristina Olmedilla and co-authors work on injection throttling, a way to hold back traffic and ease congestion in the fast nets that link supercomputers and data centers. Those nets face huge loads from science, AI, and cloud work, and a slow net can stall the whole machine.
The study reviews how such systems have grown, from faster links to added accelerators, and why the net is now a key bottleneck. It then improves throttling rules so senders pause in a smarter way. The aim is higher total throughput with less jam at peak load.
Catalog record
| CREATOR | Cristina Olmedilla, Jesus Escudero-Sahuquillo, Pedro J. Garcia, Francisco J. Quiles, Jose Duato |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-05149
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Injection-Throttling Mechanisms for Congestion Control for Data-center and Supercomputer Interconnects Cristina Olmedilla, Jesus Escudero-Sahuquillo, Pedro J. Garcia, Francisco J. Quiles, Jose Duato Abstract Over the past decade, Supercomputers and Data centers have evolved dramatically to cope with the increasing performance requirements of applications and services, such as scientific computing, generative AI, social networks or cloud services. This evolution have led these systems to incorporate high-speed networks using faster links, end nodes using multiple and dedicated accelerators, or a advancements in memory technologies to bridge the memory bottleneck. The interconnection network is a key element in these systems and it must be thoroughly designed so it is not the bottleneck of the entire system, bearing in mind the countless communication operations that generate current applications and services. Congestion is serious threat that spoils the interconnection network performance, and its effects are even more dramatic when looking at the traffic dynamics and bottlenecks generated by the communication operations mentioned above. In this vein, numerous congestion control (CC) techniques have been developed to address congestion negative effects. One popular example is Data Center Quantized Congestion Notification (DCQCN), which allows congestion detection at network switch buffers, then marking congesting packets and notifying about congestion to the sources, which finally apply injection throttling of those packets contributing to congestion. While DCQCN has been widely studied and improved, its main principles for congestion detection, notification and reaction remain largely unchanged, which is an important shortcoming considering congestion dynamics in current high-performance interconnection networks. In this paper, we revisit the DCQCN closed-loop mechanism and refine its design to leverage a more accurate congestion detection, signaling, and injection throttling, reducing control traffic overhead and avoiding unnecessary throttling of non-congesting flows. Keywords: Data-Centers, Interconnection Networks, Congestion Control, Injection Throttling, DCQCN. 1 1 footnotetext: Department of Computing Systems, Universidad de Castilla-La Mancha, Spain. Email: [email protected] 2 2 footnotetext: Royal Spanish Academy of Sciences, Spain I Motivation Supercomputers and data-centers (DCs) have evolved substantially over the past decade, and they will continue to expand in the near future [ 6 ] , as we move from the Exascale era towards Zettascale, characterized by heterogeneous architectures, cloud-native workflows, and the pervasive use of artificial intelligence (AI) across a wide range of applications. Beyond AI, domains such as large language models (LLMs), digital twins, computational fluid dynamics (CFD), climate modeling, defense, physics, and space science are also key workloads for Supercomputers or DCs. The interconnection network is the backbone of Supercomputers (SCs) and Datacenters (DCs), where thousands of computing or storage endpoints (i.e., CPUs, GPUs, HDDs, or NVMe), cooperate to run the mentioned computing- and data-hungry applications. These applications generate countless communication operations that the network must process as quickly as possible, or the network may become the system’s bottleneck. Interconnection networks are composed of interfaces (i.e., NICs) at endpoints generating traffic flows, and switches that interconnect all of them using high-speed links. These endpoints are connected within heterogeneous nodes via an intra-node network, and hundreds of these nodes are interconnected through the inter-node network. Historically, Supercomputers and DCs have diverged in their different purposes, but they have shared some characteristics; they were largely centralized, with limited cloud infrastructure; their hardware platforms mainly were homogeneous, with little architectural diversity; and their interconnection networks typically provided bandwidths in the 10–40 Gbps range on DCs and 40–100 Gbps on HPC. However, in recent years, these infrastructures have evolved to be predominantly cloud-based, with heterogeneous systems, such as edge computing and globally distributed DCs. At the same time, their interconnection networks have increased their link bandwidth to 100–800 Gbps [ 3 ] . Furthermore, the widespread adoption of GPU accelerators across both supercomputers and hyperscale DCs has effectively bridged the two ecosystems from the network architecture perspective. For example, AI/LLM training and simulation pipelines rely on collective communication patterns (e.g., all-reduce, all-to-all) that benefit from RDMA over Converged Ethernet (RoCEv2) [ 2 ] . In this context, the interconnection network must guarantee high bandwidth and low tail latency to communication operations of these applications, while minimizing energy consumption. Unfortunately, failures in network devices, aggressive power management, and, most importantly, network congestion are likely to occur as the size of current supercomputers and DCs increases, thereby spoiling the performance of the interconnection network. Specifically, congestion occurs when applications generate communication operations whose inherent traffic flows clog multiple internal network paths, overloading switch buffers. Other causes of congestion include rerouting traffic around faulty regions or lowering link speed to save power. Due to switch buffer clogging, flow-control backpressure propagates congestion from the switch where it originates (i.e., the root) back to the server end nodes generating the congesting flows. Regardless of its cause, congestion affects communication operations that do not directly contribute to network clogging (i.e., victims). This effect is known as head-of-line (HoL) blocking, which impacts network throughput, communication latency, power consumption, and more interconnection network features. Since the evolution of congestion trees (i.e., congestion dynamics) depends on each application’s traffic pattern, it is challenging to address congestion effects, such as HoL blocking or buffer hogging. Consequently, efficient congestion control (CC) techniques are required to mitigate the impact of congestion and its side effects on network performance. One of the most widely used strategies to address congestion involves monitoring the occupancy of switch buffers. When the occupancy of a buffer exceeds a given threshold, the packets populating it are considered congestion contributors. The network must notify the source end nodes generating those packets to reduce the injection rate for the traffic flows to which they belong. In this manner, congestion trees are drained from the network, thereby removing the congestion and its effects. This CC approach, known as injection throttling, has been commonly deployed in the last two decades in network technologies such as InfiniBand [ 7 ] and Ethernet [ 1 ] . As network technologies have been improved to overcome interconnection network design challenges, it is natural to expect that the CC mechanisms have evolved accordingly. For instance, the Data-Center Quantized Congestion Notification (DCQCN) [ 9 ] is an injection throttling technique designed explicitly for Ethernet-based networks and protocols (e.g., RoCEv2) and has been widely studied and deployed in current interconnection network technologies. Although vendor-specific DCQCN parameters may vary, their core philosophy persists in modern network technologies. Now it operates in both worlds: cloud AI clusters and mixed HPC installations, where GPU-driven, bursty traffic or shallow-buffer switches make precise, robust congestion control essential. Specifically, DCQCN comprises mechanisms for congestion detection, signaling, and elimination, known as Congestion Point (CP), Notification Point (NP), and Reaction Point (RP), respectively, which need to be revisited and re-designed to cope with the congestion dynamics of current applications and services. In this paper, we present several improvements to these mechanisms. Regarding CP, congestion detection is based on buffer occupancy and does not accurately differentiate between packets that contribute to congestion and those merely affected by it. We propose a more accurate congestion detection mechanism, called Enhanced Congestion Point (ECP) [ 5 , 4 ] , that eliminates the incorrect marking of victim packets. Regarding NP, note that it is required to ignore the majority of marked packets to avoid flooding the interconnect network with control traffic. Without an accurate congestion-detection