Data movement limits to frontier model training
Ege Erdil Epoch AI [email protected], David Schneider-Joseph [email protected] · 2024
Read Data movement limits to frontier model training by Ege Erdil Epoch AI [email protected], David Schneider-Joseph [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01137.txt.
Description
A 2024 arXiv research paper by Ege Erdil Epoch AI [email protected], David Schneider-Joseph [email protected]. We present a theoretical model of distributed training, and use it to analyze how far dense and sparse training runs can be scaled. Under our baseline assumptions, given a three month training duration, data movement bottlenecks begin to significantly lower hardware utilization for training runs exceeding about 10 28 10^{28} FLOP, two orders of magnitude above the largest training run to date, suggesting the arrival of fundamental barriers to scaling in three years given recent rates
Catalog record
| CREATOR | Ege Erdil Epoch AI [email protected], David Schneider-Joseph [email protected] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Data movement limits to frontier model training Ege Erdil Epoch AI [email protected], David Schneider-Joseph [email protected] Abstract We present a theoretical model of distributed training, and use it to analyze how far dense and sparse training runs can be scaled. Under our baseline assumptions, given a three month training duration, data movement bottlenecks begin to significantly lower hardware utilization for training runs exceeding about 10 28 10^{28} FLOP, two orders of magnitude above the largest training run to date, suggesting the arrival of fundamental barriers to scaling in three years given recent rates of growth. A training run exceeding about 10 31 FLOP 10^{31}\,\textrm{FLOP} is infeasible even at low utilization. However, more aggressive batch size scaling and/or shorter and fatter model shapes, if achievable, have the potential to permit much larger training runs. An interactive version of our model will shortly be accessible here . * * footnotetext: Equal contribution. 1 Introduction Scaling up neural networks and training them on more examples is crucial for good task performance ( Hestness et al. 2017 ; Kaplan et al. 2020 ; Hoffmann et al. 2022 ; Bi et al. 2024 ) , with state-of-the-art models requiring tens of thousands of GPUs 1 1 1 We focus on GPUs, but our theoretical model and findings are broadly applicable to other accelerators, and even groups of accelerators. to train in a reasonable duration. Previous work ( Huang et al. 2019 ; Rajbhandari et al. 2020 ; Lepikhin et al. 2020 ; Narayanan et al. 2021 ; Jiang et al. 2024b ) has developed practical techniques enabling the rapid scaling of the past decade ( Sevilla et al. 2022 ) . In this work, we address unexamined fundamental questions about limits to scaling in the future: Q1 Given present-day algorithms, GPUs, and interconnects, what is the biggest training run that can be performed within a fixed duration, before intra- and inter-GPU data movement starts to seriously worsen utilization or even render it impossible? Q2 How far might this limit be extended, and what algorithmic or hardware progress can achieve that? Answering these questions empirically would require millions of GPUs and large-scale engineering efforts, so we instead approach them theoretically. In doing so, we develop a simulator that can find optimal training run configurations accounting for the factors that we identify as fundamental. This gives us the answers: A1 With most current technology, GPU utilization starts to fall at \mathbf{\approx 10^{28}} floating point operations (FLOP), about three years away at recent trends ( Epoch AI 2024 ) of 4.2\times growth per year. Currently-available specialized high-bandwidth inter-node interconnects can permit training runs about two orders of magnitude larger ( \approx 10^{30}\,\textrm{FLOP} ), at which point latencies begin to worsen utilization, until reaching an absolute latency barrier at \approx 10^{31}\,\textrm{FLOP} , about seven years away. A2 Improved hardware interconnects may buy no more than two orders of magnitude in training run size, assuming technology anything like the current paradigm. Beyond that, the critical innovations must come from machine learning algorithms: The key challenge is transforming two serial dependencies — between batches and between layers — into opportunities for parallelism, by making batch sizes bigger (perhaps enabled by sparsity) and models wider and shallower. Achieving these goals may be quite difficult in practice. Data movement and latency bottlenecks limit the scale of training runs to \boldmath{10^{28}\text{ to }10^{31}} FLOP New scaling techniques and hardware innovations could push back limits to scaling Figure 1: With current technology, such as the H100 GPU and current scaling techniques, data movement bottlenecks lower hardware utilization for training runs exceeding 10^{28} FLOP, and a “latency wall” renders surpassing 10^{31} FLOP infeasible (left). However, with innovations in scaling (such as techniques to enable much larger batch sizes) or dramatic increases in network bandwidth coupled with a 10 \times reduction in inter- and intra-GPU latency, training runs can be at least a few orders of magnitude larger (right). This work is organized as follows: • Section 2 introduces a simplified model of a neural network consisting of stacked sparse linear multi-layer perceptrons that we use as the basis for our analysis throughout the paper. • Section 3 provides an overview of the four main parallelism strategies employed in distributed training—data, tensor, pipeline, and expert parallelism—and summarizes their communication costs. • Section 4 identifies the key factors that constrain distributed training, including data movement, critical batch size, and latency, and model depth. • Section 5 derives closed-form expressions for the maximum training scale under this model. • Section 6 presents the complete theoretical model accounting for all identified constraints and discusses simulation results on current hardware, demonstrating the limits to efficient scaling. We conclude by discussing the implications of our findings, and posing key open technical questions whose answers will determine the limits to large-scale model training. 2 A toy model: stacked sparse linear MLP blocks We first carefully define the class of models to be considered. Since the Transformer ( Vaswani et al. 2017 ) , including its sparse varieties ( Fedus et al. 2022 ) , is the dominant architecture today for frontier models, it seems a natural baseline. As we will show momentarily, the great majority of computation when training a Transformer occurs in its linear layers, so we adopt a simplified model consisting of these elements. This approach has the advantage that such layers also constitute the bulk of computation in many alternative architectures ( Peng et al. 2023 and Gu & Dao 2023 among others), so our model retains applicability to a wide variety of potential future algorithmic developments. However, the huge space of possible unexpected algorithmic developments precludes any watertight guarantees, should state-of-the-art architectures or learning algorithms change drastically. The formal definitions we make are as follows: a sparse linear multi-layer perceptron (MLP) block (“MLP block” for short) consists of learnable weight matrices W_{1}^{e}\in\mathbb{R}^{{d_{\text{ff}}}\times{d_{\text{model}}}} and W_{2}^{e}\in\mathbb{R}^{{d_{\text{model}}}\times{d_{\text{ff}}}} for expert indices e in the range 1\leq e\leq E , where E is the sparsity factor of the model. Given a matrix of input vectors X_{\text{in}}\in\mathbb{R}^{{d_{\text{model}}}\times b} , where b is the batch size in units of tokens, a router \rho:\penalty\ \mathbb{R}^{{d_{\text{model}}}}\to\{1,\ldots,E\} (assumed to be computationally inexpensive) chooses some expert index \rho(X_{\text{in}}^{k}) for each token index k\leq b . The sparse linear MLP f then yields an output matrix X_{\text{out}}=f(X_{\text{in}})\in\mathbb{R}^{{d_{\text{model}}}\times b} by mapping each token’s input column X_{\text{in}}^{k} to the corresponding output column X_{\text{out}}^{k} via the weight matrices corresponding to the chosen expert. If (e) indexes the set of tokens routed to expert e , then: X_{\text{out}}^{(e)}=W_{2}^{e}\Big(W_{1}^{e}X_{\text{in}}^{(e)}\Big). We optimistically assume balanced routing (Section 3.2.2 ): the same number b/E of tokens is routed to each of the E experts, so that during the forward pass every expert performs matrix multiplications of shapes ({d_{\text{ff}}},{d_{\text{model}}})\times({d_{\text{model}}},b/E)\to({d_{\text{ff}}},b/E) and then ({d_{\text{model}}},{d_{\text{ff}}})\times({d_{\text{ff}}},b/E)\to({d_{\text{model}}},b/E) , each requiring {d_{\text{model}}}{d_{\text{ff}}}b/E multiply-accumulate (MAC) operations. Across both linear layers of all E experts, the MLP block’s total arithmetic cost is 2{d_{\text{model}}}{d_{\text{ff}}}b MAC. We then assume L MLP blocks in total, so that the model’s forward pass is their composition: F(X)=f_{L}(f_{L-1}\cdots(f_{1}(X))). Across these L blocks, the model has \displaystyle N_{\text{p}}=2LE{d_{\text{model}}}{d_{\text{ff}}} (1) parameters in total and a forward pass on a batch size of b requires N_{\text{p}}b/E=2L{d_{\text{model}}}{d_{\text{ff}}}b MAC. During the backward pass, gradients of the loss \mathcal{L} must