When Scaling Fails: Network and Fabric Effects on Distributed GPU Training Performance

Dinesh Gopalan, Ratul Ali · 2026

Read When Scaling Fails: Network and Fabric Effects on Distributed GPU Training Performance by Dinesh Gopalan, Ratul Ali (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04424.txt.

Description

Gopalan and Ali trace GPU training scaling failures to network and fabric effects. Extra nodes should help as planned, yet real jobs stall early with unstable swings. Across live clusters, topology, congestion, sync, and locality often dominate performance past a few nodes. Identical software can diverge with fabric design and traffic patterns. Failure modes include sync amplification, contention from topology, and locality variance, often misread as framework flaws. Diagnostic rules from the study should improve planning and cut large scale costs.

Catalog record

CREATORDinesh Gopalan, Ratul Ali
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37452
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04424

Browse the full library · About the Tower of Babel