Scaling Intelligence: Designing Data Centers for Next-Gen Language Models Conference: 25; Nov 16–21, 2025; ISBN: 978-1-4503-XXXX-X/2025/11
Jesmin Jahan Tithi, Hanjiang Wu, Avishaii Abuhatzera, Fabrizio Petrini · 2025
Read Scaling Intelligence: Designing Data Centers for Next-Gen Language Models Conference: 25; Nov 16–21, 2025; ISBN: 978-1-4503-XXXX-X/2025/11 by Jesmin Jahan Tithi, Hanjiang Wu, Avishaii Abuhatzera, Fabrizio Petrini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15006.txt.
Introduction
Jesmin Jahan Tithi, Hanjiang Wu, Avishaii Abuhatzera, and Fabrizio Petrini design data centers for next-generation language models, in a research paper from 2025. Models with trillions of parameters strain every part of a data center, from compute to memory to networking. A joint design study can show which choices lift throughput per cost before hardware is built.
Their framework explores compute power, memory bandwidth and size, network shapes, and parallel strategies together. A flat optical network that links all nodes evenly shows strong gains in speed and scale. Sensitivity tests weigh overlap of compute and communication, hardware collectives, and wider scale-out domains. Both sparse and dense model types are covered with an analytic performance tool.
Catalog record
| CREATOR | Jesmin Jahan Tithi, Hanjiang Wu, Avishaii Abuhatzera, Fabrizio Petrini |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15006
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Intelligence: Designing Data Centers for Next-Gen Language Models Conference: 25; Nov 16–21, 2025; ISBN: 978-1-4503-XXXX-X/2025/11 Jesmin Jahan Tithi, Hanjiang Wu, Avishaii Abuhatzera, Fabrizio Petrini Abstract . The explosive growth of Large Language Models (LLMs)—such as GPT-4 with 1.8 trillion parameters—demands a radical rethinking of data center architecture to ensure scalability, efficiency, and cost-effectiveness. Our work provides a comprehensive co-design framework that jointly explores FLOPS, HBM bandwidth and capacity, multiple network topologies (two-tier vs. FullFlat optical), the size of scale-out domain, and popular parallelism/optimization strategies used in LLMs. We introduce and evaluate Fullflat network architectures, which provide uniform high-bandwidth, low-latency connectivity between all nodes, and demonstrate their transformative impact on performance and scalability. Through detailed sensitivity analyses, we quantify the benefits of overlapping compute and communication, leveraging hardware-accelerated collectives, widen scale-out domain and larger memory capacity. Our study spans both sparse (mixture of experts) and dense transformer-based LLMs, revealing how system design choices affect Model FLOPS Utilization (MFU = Model flops per token × \times Observed tokens per sec/Peak flops of the hardware) and overall throughput. For co-design study, we extended and validated a performance modeling tool capable of predicting LLM runtime within 10% of real-world measurements. Our findings offer actionable insights and a practical roadmap for designing AI data centers that can efficiently support trillion-parameter models, reduce optimization complexity, and sustain the rapid evolution of AI capabilities. 1. Introduction Future data centers must be equipped to handle the demands of multi-trillion parameter large language models (LLMs) like OpenAI’s GPT-4 with mixture of experts transformer models ( Cai et al. 2024 ) featuring 1.8 trillion parameters. These models require advanced data center designs with substantial compute power, high-bandwidth memory (HBM), and scalable, low-latency networks. The cost of training GPT-4 far exceeds that of GPT-3, underscoring the need for efficient infrastructure. The widespread adoption of LLMs across industries pushes data center co-design priorities, challenging throughput, latency, and reliability. Current systems often achieve less than 50% Model FLOPS Utilization (MFU) 1 1 1 takes into account the required operations for the forward and backward passes through the model and does not include recomputation in the measured FLOPs. ( Tazi et al. 2025 ; Shoeybi et al. 2019 ) , highlighting the need for optimization. Our co-design analysis focuses on data center architectures needed to efficiently run large-scale models like GPT-4 due to its significant impact on the infrastructure and operational demands of modern data centers. Training GPT-4 required approximately 25,000 Nvidia A100 GPUs over 90-100 days, using a dataset of 13 trillion tokens. Holding model parameters, gradients, and optimizers demands around 461 A100 GPUs, each with 80 GB of HBM, assuming 20 bytes per parameter for BF16 compute and FP32 grad accumulation ( Tazi et al. 2025 ) . Today’s AI data centers are built with clusters of GPUs that are organized into nodes with a limited number of GPUs (e.g., 8) inside each node. These GPUs are interconnected through high-speed links such as NVLink (Nvidia), XeLink (Intel) ( Jiang 2022 ) , or Infinity Fabric (AMD) ( AMD 2024 ) . Additionally, the nodes are connected via lower-bandwidth networks like Ethernet or InfiniBand throughout the data center. This setup can be viewed as a two-tier network consisting of high-bandwidth and low-bandwidth domains (HBD and LBD for short). The difference in throughput between these two is significant; for example, NVLink provides 450 GB/s, whereas Ethernet offers 50 GB/s. HBD is typically utilized for communication-intensive tensor parallelism, while LBD facilitates data and pipeline parallelism. The upcoming Nvidia Grace Blackwell GPU (i.e., GB200, GB300) enhances the HBD by expanding its capacity from 8 to 72 GPUs with an NVLink bandwidth of 900 GB/s, doubling previous capabilities. The Nvidia Rubin series plans further expansion to 144 or 576 GPUs with an NVLink bandwidth of 3600 GB/s by 2027 ( Patel et al. 2025 ) . We use “GPU" and “node" interchangeably, with “GPU" potentially replaced by TPU, CPU, XPU, or other accelerators. Optimizing data centers for large-scale models like GPT-4 is a complex and challenging task due to the numerous variables that need to be chosen and fine-tuned, from both system architecture and software algorithm perspectives. This creates a multifaceted optimization landscape. From a system architecture perspective, key factors include compute nodes, storage, network infrastructure, bandwidth, and latency. Support for high-performance collective libraries, such as Nvidia’s SHARP ( NVIDIA Corporation 2023 ; Graham et al. 2020 ) , is important for communication and synchronization. Within each node, critical parameters that directly influence LLM performance include the floating-point operations per second (FLOPS) capabilities of the GPUs across various precisions (FP64, FP32, FP16, FP8, FP4, etc.) and their efficiency at different operational sizes. The capacity, bandwidth, and transfer efficiency of the GPU’s high-bandwidth memory (HBM or Tier 1 memory) are also important metrics. Furthermore, the capacity, bandwidth, and transfer efficiency of the attached CPU’s DDR memory (or Tier 2 memory), which is often used for offloading inactive model components and checkpointing, and the bandwidth of data transfer between these memories and the GPU, are critical factors to consider ( Tazi et al. 2025 ) . On the software algorithm side, a multitude of optimization parameters require careful tuning. These include various forms of parallelism, such as data parallelism (DP), pipeline parallelism (PP), sequence parallelism (SP), context parallelism (CP), tensor parallelism (TP), expert parallelism (EP), and expert shading (ES). Additional considerations involve determining the optimal batch size and micro-batch size, the degree of pipeline interleaving, and other strategies to reduce pipeline bubbles and inefficiencies. Decisions on quantization and data types for computation, and whether to employ kernel fusion by combining activation computation with preceding computations have a performance impact. The choice of attention, such as multi-head (MHA), Multi-head Latent (MLA), multi-query (MQA), or group query (GQA) — also plays a role, as does the decision on activation recomputation (none, full, or attention-only) in determining the performance of LLM while running in a data center cluster. Deciding to do full recompute by default adds 30% overhead in the total time, where it saves significant storage needs by avoiding storing activations, thus, allowing it to run on fewer nodes. Some of the recomputation can be avoided by offloading inactive layer’s weights, activations, and optimizers to CPU/Tier 2 memory and transferring them back when needed to the GPU memory, and the decision on whether to do this or not can have an impact on the performance and storage need ( Tazi et al. 2025 ) . Additional considerations include optimizer sharding techniques such as ZERO-1, which partitions the optimizer, and ZERO-2, which extends this to include both optimizer and gradients. FSDP/ZERO-3 further expands on this by sharding the optimizer, gradients, and weights. Overlapping communication with computation in tensor parallelism (TP) and data parallelism (DP) is also important. These optimizations have specific communication requirements, and the network’s type and characteristics can affect perceived performance and scalability. The choice of collective operations for synchronizing and exchanging values across GPUs also impacts performance; for example, the all-reduce operation can be implemented as a combination of reduce-scatter and all-gather, which can be divided into chunks and interleaved with computation. When executed effectively, this interleaving can mask the data transfer cost, though it typically requires high-performance