Scalable and Efficient Intra- and Inter-node Interconnection Networks for Post-Exascale Supercomputers and Data centers

Joaquin Tarraga-Moreno, Daniel Barley, Francisco J. Andújar Muñoz, Jesus Escudero-Sahuquillo, Holger Fröning, Pedro Javier Garcia, Francisco J. Quiles, José Duato · 2025

Read Scalable and Efficient Intra- and Inter-node Interconnection Networks for Post-Exascale Supercomputers and Data centers by Joaquin Tarraga-Moreno, Daniel Barley, Francisco J. Andújar Muñoz, Jesus Escudero-Sahuquillo, Holger Fröning, Pedro Javier Garcia, Francisco J. Quiles, José Duato (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04677.txt.

Introduction

Joaquin Tarraga-Moreno leads eight authors on interconnection networks for post-exascale machines. AI, simulation, and analytics workloads push supercomputers toward mixed, tightly packed designs. More accelerators per node cut data movement but strain the links between them.

Bottlenecks appear both inside nodes and between them when components share network resources. The authors survey scalable designs for both levels of connection. High-bandwidth memory and storage trends raise the stakes for balanced communication. Their analysis targets machines where compute keeps growing faster than the wires that feed it, a central planning problem for data centers.

Catalog record

CREATORJoaquin Tarraga-Moreno, Daniel Barley, Francisco J. Andújar Muñoz, Jesus Escudero-Sahuquillo, Holger Fröning, Pedro Javier Garcia, Francisco J. Quiles, José Duato
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04677

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scalable and Efficient Intra- and Inter-node Interconnection Networks for Post-Exascale Supercomputers and Data centers Thanks: This work is supported by the Spanish Ministry of Science and Universities MCIN/AEI/10.13039/501100011033, the European Union (NextGenerationEU/PRTR) under project TED2021-130233B-C31, the Junta de Comunidades de Castilla-La Mancha and FEDER funds under the project SBPLY/21/180501/000248, the PERTE-Chip grants (UCLM Chair, TSI-069100-2023-0014) funded by the Spanish Ministry of Digital Transformation and Public Service, and the Universidad de Castilla-La Mancha under project 2023-GRIN-34056. Joaquin Tarraga-Moreno, Daniel Barley, Francisco J. Andújar Muñoz, Jesus Escudero-Sahuquillo, Holger Fröning, Pedro Javier Garcia, Francisco J. Quiles, José Duato Abstract The rapid growth of data-intensive applications such as generative AI, scientific simulations, and large-scale analytics is driving modern supercomputers and data centers toward increasingly heterogeneous and tightly integrated architectures. These systems combine powerful CPUs and accelerators with emerging high-bandwidth memory and storage technologies to reduce data movement and improve computational efficiency. However, as the number of accelerators per node increases, communication bottlenecks emerge both within and between nodes, particularly when network resources are shared among heterogeneous components. This paper analyzes the scalability challenges of such architectures and evaluates communication strategies that address the growing bandwidth and latency demands of accelerator-rich environments. We focus on the NVIDIA DGX GH200 system, which employs Grace Hopper superchips interconnected through a two-level NVLink-based slimmed fat-tree topology. Using traffic simulations under random all-to-all workloads, we evaluate multiple system configurations and compare them against reference InfiniBand (IB-NDR400) networks. Results show that the DGX GH200 interconnect achieves up to 450 Tbps of throughput—substantially outperforming traditional RLFT networks. These findings highlight the effectiveness of dedicated accelerator networks in overcoming communication bottlenecks in next-generation high-performance computing systems. Keywords: High-performance interconnection networks, Intra-node communication, accelerators, and GPUs. I Introduction Modern supercomputers and data centers are composed of thousands of interconnected server nodes that continuously expand their computing, memory, and storage capabilities to satisfy the growing demands of data-intensive applications such as scientific simulations, generative AI models, and large-scale social networks. While the demand for computational power is largely being met through increasingly powerful processors and specialized accelerators (e.g., GPUs and TPUs), emerging memory and storage technologies—such as 3D-stacked high-bandwidth memory (HBM), Non-Volatile Memory Express (NVMe), and Storage Class Memory (SCM)—are revolutionizing the architecture of these systems. By providing higher bandwidth and capacity, these technologies enable unprecedented volumes of data to be stored closer to the processing units, significantly reducing data movement and inter-node communication latency. As a result, more computation can be performed locally and efficiently within each server node. To keep pace with these developments, modern supercomputers are evolving both by scaling up—increasing the computational density within individual nodes—and by scaling out—expanding the number of interconnected nodes. A key challenge in this evolution is the growing need for faster intra- and inter-node communication, particularly among accelerators that frequently exchange large volumes of data. Companies such as NVIDIA are addressing this challenge by developing architectures that feature high-bandwidth, low-latency communication fabrics dedicated to tightly coupled accelerators, improving performance for workloads that rely on rapid local data exchange. However, as systems continue to scale, relying solely on a shared network infrastructure for all components can create communication bottlenecks. To alleviate this, modern cluster designs are increasingly adopting hierarchical or multi-tiered networking architectures, where dedicated high-speed networks interconnect specific components—such as accelerators—independently from the general-purpose system network. While this approach improves performance isolation and reduces contention, it also introduces significant complexity and cost, as it requires additional links, switches, and network management resources. At the node level, heterogeneous architectures are also becoming the norm, with systems integrating multiple types of accelerators to optimize different workloads. For instance, the JUPITER supercomputer [ 1 ] (#4 of Top500 list) incorporates four GPUs per compute node to achieve higher on-node parallelism and throughput. However, this scale-up approach presents new challenges: as the number of accelerators per node increases, so does the volume of intra-node and inter-node traffic. If the network interface controller (NIC) remains a single shared communication channel, it may become a performance bottleneck, limiting overall scalability and efficiency. This work addresses one of the main challenges in supercomputers and data centers: how do accelerators (such as GPUs, TPUs, and others) communicate with each other? We analyze several real-world systems, including the NVIDIA DGX GH200, to study their communication architectures and propose new intra- and inter-node network topologies aimed at improving overall system performance. II Background This section provides background details on a real system architecture proposed by a company, such as NVIDIA. Moreover, we analyze slimmed fat-tree topologies and their routing. II-A DGX GH200 system The DGX GH200 system [ 2 ] proposed an AI-focused system using the NVIDIA Grace Hopper GH200 superchips [ 3 ] architecture. NVIDIA GH200 Grace Hopper Superchip proposed an unified CPU–GPU architecture that merges an Arm-based NVIDIA Grace CPU with an NVIDIA Hopper GPU into a single package. The two components are connected via NVLink-C2C [ 4 ] , a high-bandwidth, low-latency, and memory-coherent interface delivering 900 GB/s of bidirectional bandwidth—seven times faster than PCIe Gen5. This superchip architecture combines up to 72 Arm Neoverse V2 CPU cores, 96 GB of HBM3 GPU memory ( 4 TB/s bandwidth), and up to 480 GB of LPDDR5X system memory ( 500 GB/s bandwidth), forming a heterogeneous computing unit optimized for AI, HPC, and data analytics workloads. The unified memory access model allows both CPU and GPU to share and access each other’s memory coherently, dramatically reducing data movement overhead. To scale beyond a single GH200 superchip, NVIDIA employs the NVLink Switch System [ 5 ] , powered by fourth-generation NVLink and third-generation NVSwitch ASICs. Each NVLink Switch provides 25.6 Tb/s of full-duplex bandwidth, exposing 128 fourth-generation NVLink ports. The NVLink Switch System forms a two-level, non-blocking slimmed fat-tree topology connecting up to 256 GH200 superchips. At the first level, groups of eight Grace Hopper modules are interconnected within a chassis by three NVLink Switch trays, providing 3.6 TB/s of intra-chassis bisection bandwidth. At the second level, 36 NVLink switches interconnect 32 of these chassis to form the complete DGX GH200 fabric, offering 115.2 TB/s of bisection bandwidth—over nine times higher than the bandwidth of an NDR400 InfiniBand fabric. This topology ensures that every GPU in the system can access the memory of any other GPU or CPU with minimal latency, creating a single, unified high-speed memory space across all 256 superchips. II-B Slimmed fat-tree Fat-tree topologies [ 6 ] are a class of hierarchical interconnection networks widely used in high-performance computing (HPC) and large-scale data centers due to their scalability, high throughput, and path redundancy. The fat-tree was designed as a universal network providing the same bandwidth at all levels of the hierarchy by extending links toward the root of the tree to avoid bottlenecks and achieve full bisection bandwidth across all communication paths. Over the years, several generalizations of the original fat-tree have been proposed, including k-ary n-trees, generalized fat-trees (GFTs), and extended generalized fat-trees (XGFTs). These models provide a flexible mathematical framework to represent multilevel switch networks with configurable degrees of connectivity and oversubscription. Fat-trees have been implemented in many supercomputers and commercial interconnects such as InfiniBand due to their deterministic routing, non-blocking behavior, and hardware efficiency. Because traditional fat-trees aim for

Read the complete text