Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving

Junsun Choi, Sam Son, Sunjin Choi, Hansung Kim, Yakun Sophia Shao, Scott Shenker, Sylvia Ratnasamy, Borivoje Nikolic · 2026

Read Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving by Junsun Choi, Sam Son, Sunjin Choi, Hansung Kim, Yakun Sophia Shao, Scott Shenker, Sylvia Ratnasamy, Borivoje Nikolic (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00254.txt.

Introduction

Junsun Choi and 7 co-authors ask whether the costly networks inside AI data centers are really needed for serving mixture-of-experts language models. In these models, much of the running time goes to machines talking to each other, and the industry has answered with expensive high-bandwidth scale-up networks. The authors compare four hardware layouts on cost and performance together.

Cheaper switchless layouts beat the scale-up design on cost-effectiveness in every scenario they explored, in some cases by more than half. A three-dimensional full-mesh layout gave the best balance of speed and cost. They also find today's scale-up links carry more bandwidth than needed, since trimming that bandwidth raised throughput per unit of cost. The advantage of switchless designs looks set to persist in coming hardware generations.

Catalog record

CREATORJunsun Choi, Sam Son, Sunjin Choi, Hansung Kim, Yakun Sophia Shao, Scott Shenker, Sylvia Ratnasamy, Borivoje Nikolic
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00254

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving Junsun Choi, Sam Son, Sunjin Choi, Hansung Kim, Yakun Sophia Shao, Scott Shenker, Sylvia Ratnasamy, Borivoje Nikolic Abstract Mixture-of-experts (MoE) architectures have turned LLM serving into a cluster-scale workload in which communication consumes a considerable portion of LLM serving runtime. This has prompted industry to invest heavily in expensive high-bandwidth scale-up networks. We question whether such costly infrastructure is strictly necessary. We present the first systematic cross-layer analysis of network cost-effectiveness for MoE LLM serving, comparing four representative XPU (e.g., GPU/TPU) topologies (scale-up, scale-out, 3D torus, and 3D full-mesh). We find that lower-cost switchless topologies are more cost-effective than the scale-up topology across all serving scenarios explored, improving cost-effectiveness by 20.6–56.2%. In particular, the 3D full-mesh topology is Pareto-optimal in terms of the performance-cost tradeoff. We also find that current scale-up link bandwidths are over-provisioned: reducing the link bandwidth improves throughput per cost by up to 27%. A forward-looking analysis of upcoming GPU generations indicates that the cost-performance advantage of switchless networks will likely persist. * * footnotetext: Co-first authors 1 Introduction Figure 1 : A lower-performance but low-cost network can be a better choice for an MoE serving cluster. Left: A torus cluster may provide lower throughput (smaller Y value) than a scale-up cluster, but its lower cost (smaller X value) allows deploying more clusters, reducing cost at iso-throughput. Right: With computation-communication overlap, the lower-cost topology is the better choice if the exposed communication time (ECT) is zero in both clusters. As the adoption of LLMs grows, from code generation [ 2 ] to scientific discovery [ 43 ] , LLM serving is becoming increasingly important. OpenAI serves more than 250 million tokens per second [ 45 ] . Hyperscalers are investing hundreds of billions of dollars in GPU clusters to meet this demand [ 63 ] . With the rise of mixture-of-experts (MoE) architectures in LLMs, LLM serving has recently evolved from a single-node workload into a cluster-scale workload. The MoE architecture keeps per-token computation relatively bounded, enabling model capacity to scale to even trillions of parameters. This rapid model size growth outpaces single-device memory capacity growth, exacerbating the “memory wall" problem [ 17 ] , forcing the number of XPUs (chips to run AI models on, such as GPUs or TPUs) required for serving to reach cluster-scale. As serving becomes parallelized, communication accounts for a considerable portion of the end-to-end runtime of LLM serving [ 24 , 69 ] . For example, communication consumes up to 59% of runtime in a 96-GPU MoE serving setup [ 56 ] . Industry has reacted by building ever-larger "scale-up" networks [ 21 , 7 ] , enlarging the high-bandwidth domain to minimize the communication time, but also inflating cluster costs with numerous expensive high-capacity switches. Is this strategy of increasing network investment necessary? Can a lower-cost network still be sufficiently performant for MoE LLM serving? We present such a scenario in Figure 1 . An alternative is to use a lower-cost switchless topology like a torus or lower the link bandwidth, which could be slower but still be preferable, since the saved cost allows deploying additional clusters to match the scale-up throughput (Left). Thus, network designs should be evaluated by their throughput per cost when cluster expansion is feasible. Also, because LLM serving allows aggressive overlap of communication and computation [ 16 , 75 ] , exposed communication time (ECT) is the right metric for comparing networks. We need only enough network capacity to achieve ECT \approx 0: any less hurts performance, and any more wastes investment (Right). We refer to a network that attains this sweet spot as a “cost-effective” network. In this paper, we focus on estimating the cost-effectiveness of different network designs. Determining the sweet spot is challenging because it requires a holistic view of performance. That is, one must look beyond network performance in isolation and instead understand both compute and communication times as well as how they overlap. For this, we must understand how key system parameters such as memory size, context length, and service-level objective (SLO) requirements jointly impact performance. Furthermore, the cost-performance tradeoff is sensitive to software optimizations like speculative decoding (SD) and dual-batch overlap (DBO) (§ 2.3 ) and hence must also be captured by our evaluation. Figure 2 : Network topologies of current XPU clusters We address these challenges for serving via a holistic, cross-layer methodology that systematically models communication time, compute time, and cluster cost for MoE LLM inference by carefully combining real-world benchmarks with simulation. We apply our methodology to four representative XPU network topologies (scale-up, scale-out, 3D torus, and 3D full-mesh) [ 21 , 76 , 35 ] (Figure 2 ), analyzing performance-cost tradeoffs in realistic serving scenarios with SD and DBO, and across a range of configurations for link bandwidth and latency. Our key findings are as follows: 1) Current systems are not cost-effective: We find that in scale-up systems, the current link-bandwidth provisioning lies beyond the sweet spot for cost-effectiveness: reducing link bandwidth alone improves throughput per cost by up to 27% (§ 4.2 ). When combined with the topology comparison, our analysis shows lower-cost switchless topologies, such as 3D torus and full-mesh, are more cost-effective than scale-up and scale-out networks across all LLM serving scenarios we explore, improving cost-effectiveness by 20.6–56.2%. In particular, 3D full-mesh, despite being less widely adopted than other topologies, provides the best overall tradeoff (§ 4.3 ,§ 4.4 ). This trend arises because, under a short context length and a more relaxed SLO (higher TPOT), DBO is effective such that the exposed communication time can be eliminated. SD further expands the regime in which DBO is effective, strengthening the cost-effectiveness of lower-cost switchless networks. Under a long context length or a tight SLO, the small maximum achievable batch size leads to a smaller message size, removing the performance advantage of scale-up over torus and full-mesh. 2) Switchless topologies will remain more cost-effective than switch-based topologies in the next generation: Through a forward-looking analysis that considers projected compute and memory scaling over the next two GPU generations (NVIDIA Blackwell and Rubin), we show that the cost-effectiveness advantage of the switchless networks is likely to persist in the near future for most serving scenarios (§ 4.5 ). One caveat, however, is that compute performance and memory bandwidth are expected to scale faster than interconnect bandwidth in the Rubin generation (Table 5 ). If this trend continues, bandwidth provisioning may be insufficient to sustain the advantages of switchless networks. Lowering the software and protocol overhead, alongside bandwidth scaling, helps maintain the advantage of switchless topologies in the next generation. To our knowledge, we are the first to conduct a systematic analysis of the cost-effectiveness of different network designs for MoE LLM serving, taking communication time, compute time, software optimizations, and cluster cost into consideration. A line of work on network comparison [ 39 , 65 , 49 , 12 , 48 , 32 , 38 ] optimizes the collective communication time (CCT) of LLMs, but not ECT. Other work has looked at cross-layer XPU network design in the context of training [ 47 , 35 , 62 , 27 , 67 ] , but serving presents

Read the complete text