Leveraging InfiniBand Controller to Configure Deadlock-Free Routing Engines for Dragonflies
German Maglione-Mathey, Jesus Escudero-Sahuquillo, Pedro Javier Garcia, Francisco J. Quiles, Eitan Zahavi · 2025
Read Leveraging InfiniBand Controller to Configure Deadlock-Free Routing Engines for Dragonflies by German Maglione-Mathey, Jesus Escudero-Sahuquillo, Pedro Javier Garcia, Francisco J. Quiles, Eitan Zahavi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01214.txt.
Introduction
German Maglione-Mathey and four co-authors tackle routing in high speed computer networks in a 2025 paper. The networks use a layout called Dragonfly, common in large parallel machines. A known routing method avoids deadlocks, where messages block each other forever, but the standard network controller did not offer it.
The authors give a plain method to add that routing method as an engine in the controller, with full detail on the setup it needs. They test the new engine on a real cluster and in simulation. Results confirm it works and show how its speed and demands compare with engines the controller already offers.
Catalog record
| CREATOR | German Maglione-Mathey, Jesus Escudero-Sahuquillo, Pedro Javier Garcia, Francisco J. Quiles, Eitan Zahavi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-01214
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging InfiniBand Controller to Configure Deadlock-Free Routing Engines for Dragonflies German Maglione-Mathey, Jesus Escudero-Sahuquillo, Pedro Javier Garcia, Francisco J. Quiles, Eitan Zahavi Abstract The Dragonfly topology is currently one of the most popular network topologies in high-performance parallel systems. The interconnection networks of many of these systems are built from components based on the InfiniBand specification. However, due to some constraints in this specification, the available versions of the InfiniBand network controller (OpenSM) do not include routing engines based on some popular deadlock-free routing algorithms proposed theoretically for Dragonflies, such as the one proposed by Kim and Dally based on Virtual-Channel shifting. In this paper we propose a straightforward method to integrate this routing algorithm in OpenSM as a routing engine, explaining in detail the configuration required to support it. We also provide experiment results, obtained both from a real InfiniBand-based cluster and from simulation, to validate the new routing engine and to compare its performance and requirements against other routing engines currently available in OpenSM. Journal: Journal on Parallel and Distributed Computing 1 Motivation High-Performance Computing (HPC) systems, as well as Datacenters, consist of a high number of computing and storage endnodes interconnected by means of a network. Given the increasingly growing communication requirements of the services and applications supported by these systems, the performance demanded to the interconnection network (mainly high throughput and low latency) augments accordingly. Hence, it is essential to optimize the design and configuration of the interconnection network in order to achieve the desired performance required by the system services and applications. One of the most important design aspects of any interconnection network is its topology, i.e. the pattern followed to interconnect the endnodes of the system. In that sense, Dragonfly topologies [ 1 ] have become very popular in the last years because of their low diameter, path diversity, high bisection bandwidth, and good performance/cost ratio. In addition, they offer a good scalability due to their hierarchical structure. Thanks to these advantages, Dragonfly topologies have been configured in several real systems such as the IBM PERCS [ 2 ] , the Cray XC series [ 3 ] or the Niagara (the Canada’s fastest supercomputer) [ 4 ] , which implements a Dragonfly + architecture [ 5 ] . However, the interconnection pattern of Dragonfly topologies contains physical cycles that may lead to cyclic channel dependencies and so to traffic deadlocks. When a packet is allowed by the routing algorithm to traverse two channels c_{n} and c_{m} , in that order, to reach its destination, this creates a direct dependency from c_{n} to c_{m} . The set of all direct dependencies that packets create, starting from all sources to all destinations, forms a channel dependency graph [ 6 ] , where each vertex represents a channel and an edge represents a dependency. Cyclic dependencies are formed between two or more channels that depend on each other, either directly or indirectly. This situation may lead to a cycle of packets waiting for one another to release resources (i.e. channels) in a deadlock configuration [ 7 ] , and hence remain blocked indefinitely. Hence it is mandatory that the routing algorithms used in Dragonflies are designed to guarantee deadlock freedom. For that purpose, the routing algorithms, either deterministic or adaptive, usually considered as suitable for Dragonflies prevent potential cyclic dependencies among channels [ 6 ] by following one out of three potential approaches. The first approach uses Virtual Channels (VCs) [ 6 ] as escape ways, i.e. the VC assigned to a packet may vary along their route so that no cyclic channel dependencies appear in the network for a given VC. In the second approach, the routes that follows the packets in the network are grouped into disjoint sets, and each set is mapped to an exclusive VC, all this according to some algorithm that guarantees that not all the routes that form any possible cycle belong to the same set (i.e. that not all of the routes forming a cycle share the same VC). This second approach is in general known as “layered routing” [ 8 ] . The third approach restricts routes so that the allowed ones never form a cycle. The use in Dragonflies of routing algorithms based on the first approach was proposed at the same time as this topology by Kim and Dally [ 1 ] . Specifically, a deterministic minimal routing algorithm that requires 2 VCs to provide deadlock freedom was proposed, as well as an oblivious non-minimal routing algorithm that requires 3 VCs, the latter being an adaptation of the Valiant’s routing algorithm [ 9 ] . In addition, it was also suggested the use of both minimal and non-minimal routes according to the adaptive routing algorithm proposed in [ 1 , 10 ] , which requires 3 VCs to provide deadlock freedom. However, these algorithms, similarly to others providing deadlock freedom through the same approach (e.g. the ones proposed in [ 11 , 12 , 13 ] ), are not easy to implement in real systems due to the requirement of “shifting” the VC assigned to some packets. In particular, this “VC shifting” is difficult to achieve in networks built from components based on the InfiniBand architecture [ 14 ] (see Section 2.3 ), that is currently one of the most widely used network technology in HPC systems [ 15 ] . In that sense, although VCs can be emulated through the InfiniBand Virtual Lanes (VLs), some restrictions in the InfiniBand specification [ 14 ] (see Section 3 ) make it complex to shift the VL assigned to a packet along its route, as required by the routing algorithms that follow the first approach for deadlocks prevention. For that reason, the deadlock-free routing algorithms that have been implemented as routing engines in the InfiniBand control software (OpenSM [ 16 ] ) follow mainly the second or third approaches mentioned above. The available routing engines suitable for Dragonflies that follow the second approach (i.e. layered routing) are LASH [ 8 ] , DFSSSP [ 17 ] and D3R [ 18 ] , the former two being actually topology agnostic algorithms while the latter having been specially designed for Dragonflies. Regarding the third approach, the classical, topology-agnostic Up/Down algorithm is also available as a routing engine (UPDN) in OpenSM [ 19 ] , which can be also used in Dragonflies. As far as we know, the only deadlock-free routing algorithm following the first approach (i.e. requiring VL shifting) and suitable for Dragonflies that has been implemented as a IB routing engine is DF-DN [ 20 ] . This routing engine actually was proposed in general for low-diameter topologies, and it is based on a non-uniform configuration of the tables that map the packets Service Level (SL, see Section 2.3 ) to VLs, which indeed introduces some degree of complexity in the network configuration. It is worth clarifying that all the routing engines mentioned in the former paragraph are deterministic, since implementing adaptive routing algorithms in InfiniBand presents very difficult or even unsurmountable problems. It is worth clarifying also that DFSSSP and LASH routing engines are strictly minimal-path and UPDN may provide non-minimal routes. Nevertheless, D3R is minimal if we consider