Extreme-Scale Interconnection Networks
Alejandro Cano, Cristina Brinza, Cristóbal Camarero, Carmen Martínez, Ramón Beivide · 2026
Read Extreme-Scale Interconnection Networks by Alejandro Cano, Cristina Brinza, Cristóbal Camarero, Carmen Martínez, Ramón Beivide (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26960.txt.
Introduction
Cano and four colleagues examine leaf-spine network topologies for extreme-scale data centers, starting from two Fat-Tree alternatives: the Orthogonal Fat-Tree and the Random Folded Clos. Their Multipass Random Leaf-Spine networks inherit both designs' strengths and beat Fat-Trees on throughput and flexibility. With non-minimal routing strategies, simulations show a 50 percent speedup over Fat-Tree and 100 percent over Dragonfly on an All2All collective of 100,000 endpoints.
Catalog record
| CREATOR | Alejandro Cano, Cristina Brinza, Cristóbal Camarero, Carmen Martínez, Ramón Beivide |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57127 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26960
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Extreme-Scale Interconnection Networks Alejandro Cano, Cristina Brinza, Cristóbal Camarero, Carmen Martínez, Ramón Beivide Abstract Extreme-scale data centers are the backbone of next-generation computing, enabling breakthroughs in science, artificial intelligence, and global innovation through unprecedented processing power and scalability. This work examines leaf-spine network topologies that offer extreme scalability—connecting a vast number of endpoints—while delivering strong performance at low cost. It takes as a starting point two alternatives to the widely used Fat-Tree topology: the Orthogonal Fat-Tree and the Random Folded Clos. The resulting Multipass Random Leaf-Spine (MRLS) networks inherit their advantages and surpass Fat-Trees in both throughput and flexibility. To fully leverage the topological properties of these networks, various non-minimal routing strategies are considered. An exhaustive evaluation using an interconnection network simulator provides insight into the trade-offs and scalability of these topologies under realistic conditions, positioning them as a promising solution for extreme-scale systems. The MRLS achieves a 50% speedup against a Fat-Tree for an All2All collective comprising 100k endpoints, and 100% against Dragonfly networks for the same collective. 1 Introduction As workload demands in Data Center (DC) and High-Performance Computing (HPC) systems continue to increase, the scale of network interconnects also expands. This growth directly impacts the achievable performance of the overall system. Extreme-scale DCs interconnect hundreds of thousands of endpoints, with projections of their upgrade to millions [ 1 , 2 , 3 ] . A significant proportion of the interconnects used nowadays are indirect networks such as the Fat-Tree (FT) or Folded Clos [ 4 ] . In an indirect network (IN), switches are organized into levels. The lowest level consists of leaf switches , which connect directly to endpoints and to switches in the level above. The highest level consists of root switches (or spines if it’s a two-level network), which connect only to the switches below them. Communication between endpoints is indirect , meaning traffic must traverse intermediate switches that are not connected to any endpoint. In contrast, in direct networks (DN) such as Dragonfly (DF) [ 5 ] , there is no distinction among switches, and all of them have endpoints connected. The FT is a particularly notable indirect network for being rearrangeably non-blocking, meaning it can support communication between any pair of nodes using disjoint paths. This implies there is no interference between multiple transmissions when employing circuit switching. However, the implementation cost of FT networks is substantially higher compared to other alternatives, prompting the exploration of more scalable and cost-effective solutions. For extreme-scale systems, where the network can account for a significant fraction of the system cost, minimizing it is crucial, even if it poses additional challenges. Among scalable 2-level network topologies, the Orthogonal Fat-Tree (OFT) is a prominent candidate [ 6 ] . Figure 1 shows the smallest one. While it offers much greater scalability than FT, its performance degrades significantly when subjected to non-uniform traffic. To address this, non-minimal routing strategies are often employed to improve throughput and load balancing [ 7 ] . However, a key limitation of the OFT lies in its rigid construction requirements: it can only be built for very specific parameter values. This constraint limits both the applicability and expandability of the topology, making it difficult to adapt to the desired system size. Figure 1 : Orthogonal Fat Tree with parameter q=2 . It has n=14 leaf switches and k=3 up-links at the leaf switches. Figure 2 : MRLS with n=14 leaf switches and k=3 up-links at the leaf switches. Highlighted a pair of nodes at distance 4 and one path. The Random Folded Clos (RFC) [ 8 ] offers an alternative topology designed to balance performance and scalability while facilitating flexible system expansion. Unlike the strictly defined interconnection pattern of the OFT, the RFC employs a random interconnection pattern between switch levels. Figure 2 depicts a 2-level, randomly interconnected network featuring the same number of switches and endpoints as the OFT shown in Figure 1 . However, this topology deviates from the standard RFC, as it lacks inherent up-down connectivity and allows minimal path lengths of up to four hops. Consequently, we introduce this distinct topology which we call Multipass Random Leaf-Spine (MRLS), alongside a corresponding routing mechanism. This work investigates such networks and evaluates their suitability for extreme-scale systems. In particular, our contributions are as follows: 1. The proposal of MRLS, a novel network topology, alongside a routing method for randomly wired indirect networks. 2. A comprehensive analysis of the architectural benefits of MRLS, demonstrating key topological advantages such as fine-grained scalability and enhanced routing efficiency. 3. An extensive performance evaluation comparing MRLS networks against state-of-the-art indirect and direct topologies such as Dragonfly, including large-scale simulations supporting up to 100,000 endpoints. The structure of the paper is as follows. Section 2 introduces the notation for topologies and formulae for performance and cost. Section 3 elaborates on the limitations of current topologies. In Section 4 , the MRLS proposal is formally introduced and studied. Section 5 describes the workloads for simulation and selected scenarios. The simulation results are shown and analyzed in Section 6 . Section 7 concludes the paper. 2 Background This section establishes the basic concepts and notation used in the paper. We begin by defining the network models and standard topologies, followed by a formal analysis of the relationships between topological distances, performance, and cost. Table 1 summarizes the key notation employed throughout the article. 2.1 Topologies Table 1 : Notation throughout the article. S Total number of endpoints. N Total number of switches. M Total number of bidirectional links. R Switch radix. A Average distance between leaf switches in an IN, or between vertices in a DN. D Diameter, the maximum distance between leaf switches. d(x,y) Distance from x to y . \Theta Estimator of the load per endpoint. l Number of switch levels. h Height of a multistage network, h=l-1 . N_{i} Number of switches in the level i . u Number of upward ports of a switch in an IN. d Number of downward ports of a switch in an IN. f Thickness factor of the switches in an IN. q A power of prime, employed in OFT (cf. [ 9 , 10 ] ). The topology of a network describes the interconnection pattern between endpoints and switches. In many high-performance systems, each endpoint possesses a single network interface and is connected to one switch. This allows us to omit the endpoints and abstract the topology as a graph where vertices represent switches and edges represent the physical links between them. Although this abstraction simplifies the model, it is robust. Specialized architectures like BCube [ 11 ] (multi-interface endpoints) or GPU-dense nodes like those in the Frontier supercomputer (multiple NICs per endpoint) technically deviate from this simple model. However, these cases represent only a constant-factor increase in connectivity. Therefore, we proceed with the switch-level graph model, noting that extensions to multi-interface scenarios are straightforward. Next, FTs and other more cost-effective topologies are introduced. 2.1.1 The non-blocking Fat-Tree The most prevalent topology in modern DCs and HPC systems is the Fat-Tree [ 12 ] , which is essentially a multistage folded Clos network [ 13 ] . To ensure non-blocking behavior, standard Fat-Trees are built such that the aggregate