Don’t be lazy: CompleteP enables compute-efficient deep transformers
Nolan DeyCerebras Systems, Bin Claire ZhangCerebras SystemsLorenzo NociETH ZurichPrinceton UniversityMufan LiPrinceton UniversityBlake BordelonHarvard UniversityShane BergsmaCerebras SystemsCengiz PehlevanHarvard UniversityKempner InstituteBoris HaninPrinceton UniversityJoel HestnessCerebras Systems · 2025
Read Don’t be lazy: CompleteP enables compute-efficient deep transformers by Nolan DeyCerebras Systems, Bin Claire ZhangCerebras SystemsLorenzo NociETH ZurichPrinceton UniversityMufan LiPrinceton UniversityBlake BordelonHarvard UniversityShane BergsmaCerebras SystemsCengiz PehlevanHarvard UniversityKempner InstituteBoris HaninPrinceton UniversityJoel HestnessCerebras Systems (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01618.txt.
Introduction
Nolan Dey and co-authors fix a scaling flaw in how giant language models are tuned. As models deepen, good settings like learning rate stop transferring, forcing costly re tuning. Worse, some setups leave layers lazy, learning only near their starting point and wasting depth. Their scheme, CompleteP, retunes every rule with size so settings carry across depth.
It also keeps every layer actively learning features, not coasting. The payoff is a wider family of efficient model shapes for varied hardware. Compute savings ran into double digits over the prior best. All trials ran on Cerebras hardware, and a minimal version is public.
Catalog record
| CREATOR | Nolan DeyCerebras Systems, Bin Claire ZhangCerebras SystemsLorenzo NociETH ZurichPrinceton UniversityMufan LiPrinceton UniversityBlake BordelonHarvard UniversityShane BergsmaCerebras SystemsCengiz PehlevanHarvard UniversityKempner InstituteBoris HaninPrinceton UniversityJoel HestnessCerebras Systems |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01618
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Don’t be lazy: CompleteP enables compute-efficient deep transformers Nolan DeyCerebras Systems, Bin Claire ZhangCerebras SystemsLorenzo NociETH ZurichPrinceton UniversityMufan LiPrinceton UniversityBlake BordelonHarvard UniversityShane BergsmaCerebras SystemsCengiz PehlevanHarvard UniversityKempner InstituteBoris HaninPrinceton UniversityJoel HestnessCerebras Systems Abstract We study compute efficiency of LLM training when using different parameterizations , i.e., rules for adjusting model and optimizer hyperparameters (HPs) as model size changes. Some parameterizations fail to transfer optimal base HPs (such as learning rate) across changes in model depth, requiring practitioners to either re-tune these HPs as they scale up (expensive), or accept sub-optimal training when re-tuning is prohibitive. Even when they achieve HP transfer, we develop theory to show parameterizations may still exist in the lazy learning regime where layers learn only features close to their linearization, preventing effective use of depth and nonlinearity. Finally, we identify and adopt the parameterization we call CompleteP that achieves both depth-wise HP transfer and non-lazy learning in all layers. CompleteP enables a wider range of model width/depth ratios to remain compute-efficient, unlocking shapes better suited for different hardware settings and operational contexts. Moreover, CompleteP enables 12-34% compute efficiency improvements over the prior state-of-the-art. Figure 1: We introduce CompleteP , which offers depth-wise HP transfer ( Left ), FLOP savings when training deep models ( Middle ), and a larger range of compute-efficient width/depth ratios ( Right ). 1 Introduction A hallmark of modern deep learning is that training larger models leads to better performance [ 1 ] . In Large Language Models (LLMs), for instance, the paradigm of pre-training larger models on larger datasets has led to remarkable results in a wide range of downstream evaluations. However, these gains come at a substantial increase in computational cost. As compute budgets grow, practitioners must navigate a complex design space to choose model width and depth, dataset size, batch size, number of training steps, and a variety of other hyperparameters (HPs) in order to find an optimal allocation of resources to minimize a pretraining objective, given a fixed compute budget [ 2 , 3 ] . The prohibitive cost of naively conducting such a search in large models can result in suboptimal HPs and hence an inefficient use of computational resources. This motivated techniques such as the maximal update parameterization (µP), which ensures that optimal HPs remain approximately constant when scaling model width [ 4 , 5 ] and enables a “tune small and train large” strategy. We refine and thoroughly compare extensions of µP for simultaneously scaling depth and width [ 6 , 7 , 8 ] . These scaling strategies define parameterizations — sets of rules for how to scale model and optimizer hyperparameters with model size. The core difference between µP and these depth-aware refinements is how they re-scale the outputs of the transformer’s [ 9 ] residual block (as a function of depth L ) before the output gets added to the residual stream variables h^{\ell} : \displaystyle\mathbf{h}^{\ell+1}=\mathbf{h}^{\ell}+L^{-\alpha}\ \mathcal{F}_{\ell}(\mathbf{h}^{\ell})\ ,\ \ell\in\{1,...,L\}, (1) where \mathcal{F}_{\ell} is the \ell ’th residual block (for transformers, these are MLP and attention blocks). The depth-dependent re-scaling factor is governed by a single parameter \alpha\in[0.5,1] , where a larger \alpha corresponds to stronger downscaling. Setting \alpha=1 gives a parameterization we call CompleteP because it is the unique \alpha ensuring complete feature learning as width and depth are scaled ( Section 6 ). Since \alpha=0.5 and \alpha=1 are the two most promising potential candidates for depth scaling [ 6 , 7 , 8 ] , we compare these two values and find \alpha=1 is consistently more compute-efficient through better HP transfer and faster pre-training ( Figure 1 ). To realize these gains, however, we must extend the parameterization to include principled re-scalings of LayerNorm (LN) and bias learning rates, AdamW’s weight decay \lambda , and AdamW’s \epsilon (Table 1 ). CompleteP is simple to implement, yet yields superior upstream and downstream performance, with gains over alternative approaches increasing with model depth. To summarize, our contributions are: • HP transfer across depth. We compare transfer of learning rate and weight initialization standard deviation across depth for the standard parameterization (SP), µP, and \alpha=\{0.5,1\} , when pre-training Pre-LN transformers ( Figure 2 , 3 ). We find only \alpha=1 enables both effective HP transfer and improved performance as model depth increases. • Optimal shape and compute efficiency. For SP, µP, and \alpha=\{0.5,1\} , we construct scaling laws over models ranging from 75 M to 1.9 B total parameters, trained at a compute-optimal frontier of 20 tokens per parameter (TPP) [ 3 ] . We use this to revisit the question of compute-efficient transformer shapes (i.e. width-to-depth ratio N\!:\!L ). The 1.9B parameter models (i.e. with 1.5B non-embedding parameters) trained with \alpha=1 achieve 11.8% and 34.4% FLOP savings over µP for optimally-shaped and 179-layer models, respectively ( Figure 4 ). • Refined desiderata for HP transfer. In Section 6 , we describe the differences between SP, µP, and \alpha\in\{0.5,1\} . Given the strong empirical evidence for HP transfer for \alpha=1 , we devise a refined set of desiderata for HP transfer that includes the notion of complete feature learning : we require that the learned representation in every layer of the model remains non-lazy [ 10 ] (i.e. non-linear) with respect to the parameter in both that layer and all earlier ones. Only \alpha=1 ensures complete feature learning, thus we name it CompleteP. • Extra HP guidelines. In Table 1 and Appendix E , we extend existing parameterizations [ 8 ] to include prescriptions for common transformer HPs (such as weight decay, bias and LayerNorm learning rates \eta , and AdamW \epsilon ) as functions of depth and width. Figure 7 shows these changes are essential for stable training with \alpha=0.5 in particular. 2 Related work Theoretical approach to HP tuning. Early theoretical approaches for choosing HPs analyzed networks at initialization, seeking to guarantee different notions of numerical stability of the first forward and backward pass [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ] . Subsequent work devised two parameterizations with consistent training dynamics at infinite width: the Neural Tangent Kernel (NTK) parameterization in which the model converges to its linearization around initialization [ 29 , 30 , 31 ] and the mean-field/µP parameterization [ 4 , 6 , 8 , 10 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 ] . Training at the simultaneous limits of width and depth was explored in [ 40 ] for fully connected networks, in [ 6 , 7 ] for vanilla ResNets and in [ 8 ] for transformers. Infinite-limit descriptions of training in the mean-field/µP parameterization are difficult to study analytically but give a clear proposal for HP transfer by requiring consistent dynamics of hidden layer representations across model scale [ 41 , 42 , 43 , 44 , 32 ] . This approach to HP transfer was taken up in [ 6 , 7 , 8 ] , which derive a family of mean-field parameterizations for ResNets indexed by \alpha\in[0.5,1] (see Table 1 ). Based on a heuristic