When is Warmstarting Effective for Scaling Language Models?

Neeratyoy Mallik 1,2,∗ Maciej Janowski 1,3,∗ Johannes Hog 1,∗ Herilalaina Rakotoarison 4 Josif Grabocka 3 Frank Hutter 5,6,1 Aaron Klein 6 1 University of Freiburg 2 Zuse School ELIZA 3 University of Technology Nuremberg 4 University of Helsinki 5 Prior Labs 6 ELLIS Institute Tübingen ∗ Equal contribution. Correspondence: {mallik,janowski,hogj}@cs.uni-freiburg.de · 2026

Read When is Warmstarting Effective for Scaling Language Models? by Neeratyoy Mallik 1,2,∗ Maciej Janowski 1,3,∗ Johannes Hog 1,∗ Herilalaina Rakotoarison 4 Josif Grabocka 3 Frank Hutter 5,6,1 Aaron Klein 6 1 University of Freiburg 2 Zuse School ELIZA 3 University of Technology Nuremberg 4 University of Helsinki 5 Prior Labs 6 ELLIS Institute Tübingen ∗ Equal contribution. Correspondence: {mallik,janowski,hogj}@cs.uni-freiburg.de (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13405.txt.

Introduction

Neeratyoy Mallik and six colleagues ask when growing a model from an existing checkpoint saves effort. The field has stressed too much keeping the smaller model's results when it grows. That stress limits new designs, and the link between growth and hyperparameters is little studied. Their tests show keeping first results is not needed for strong final results. Simple growth ways can beat fancier warmstart methods, and doubling the model best speeds training. An upper bound on growth exists beyond which training from scratch wins.

Catalog record

CREATORNeeratyoy Mallik 1,2,∗ Maciej Janowski 1,3,∗ Johannes Hog 1,∗ Herilalaina Rakotoarison 4 Josif Grabocka 3 Frank Hutter 5,6,1 Aaron Klein 6 1 University of Freiburg 2 Zuse School ELIZA 3 University of Technology Nuremberg 4 University of Helsinki 5 Prior Labs 6 ELLIS Institute Tübingen ∗ Equal contribution. Correspondence: {mallik,janowski,hogj}@cs.uni-freiburg.de
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88485
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When is Warmstarting Effective for Scaling Language Models? Neeratyoy Mallik 1,2,∗ Maciej Janowski 1,3,∗ Johannes Hog 1,∗ Herilalaina Rakotoarison 4 Josif Grabocka 3 Frank Hutter 5,6,1 Aaron Klein 6 1 University of Freiburg 2 Zuse School ELIZA 3 University of Technology Nuremberg 4 University of Helsinki 5 Prior Labs 6 ELLIS Institute Tübingen ∗ Equal contribution. Correspondence: {mallik,janowski,hogj}@cs.uni-freiburg.de Abstract Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited practical adoption in large-scale training. We attribute this to two underexplored factors: (1) an overemphasis on preserving the smaller model’s performance at initialization, which constrains operator design for new architectures, and (2) insufficient analysis of how growth interacts with hyperparameters and scaling behavior, compounded by inconsistent growth factors across the literature. We show that preserving the base model’s initial post-growth performance is not necessary for strong final performance, and that simple, architecture-agnostic growth strategies can outperform more complex warmstarting operators. Crucially, we empirically identify an upper bound on the growth factor g g beyond which training from scratch is more efficient. We observe this across multiple ablation setups. Notably, this limit is also present, but unreported, in prior published results. Across our experiments on dense MLPs and dense language models, we find that a 2 × 2\times growth factor is the most reliable in yielding convergence speedups, with gains most pronounced under 20 tokens/parameter budgets and diminishing as budget increases. We fit scaling laws over these observations to provide predictive guidance for practitioners deciding when and how much to grow. Together, our analysis provides practical guidelines and empirical limits for model growth. 1 Introduction Training state-of-the-art deep learning models requires significant computational resources. As models scale from millions to billions of parameters, practitioners face an important question: can we reuse the compute invested in training smaller models when scaling up? Model growth or warmstarting ( WS ), i.e., initializing larger models from trained smaller checkpoints, offers potential efficiency gains, yet remains largely absent from production pipelines despite substantial research ( Gong et al. 2019 ; Chen et al. 2021 ; Rae et al. 2021 ; Shen et al. 2022 ; Wang et al. 2023a ; Wang et al. 2023b ; Deng et al. 2023 ; Samragh et al. 2024 ; Du et al. 2024 ; Yu et al. 2026 ) . Deep Learning (DL) success has been built on pushing the loss frontier across tasks, applications, and data modalities ( Krizhevsky et al. 2012 ; He et al. 2016 ; Vaswani et al. 2017 ; Brown et al. 2020 ; Radford et al. 2021 ; Rombach et al. 2022 ; OpenAI 2024 ) . As training efficacy improves, processing larger datasets drives corresponding increases in model capacity ( Alabdulmohsin et al. 2023 ) . Current research on frontier models builds on seminal scaling law studies demonstrating the necessity of simultaneously growing both model size and data processed ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) . This co-scaling question continues to evolve across different constraints, modalities, and architectures, with the traditional goal of pushing the loss-frontier now complemented by optimizing the compute-optimal frontier ( Anagnostidis et al. 2024 ) . In practice, these considerations translate to decisions about resource efficiency ( Wan et al. 2023 ) . While WS has been periodically explored across architectures and paradigms, it has not become standard practice in training pipelines, often remaining an academic topic rather than a common choice in DL pipelines ( Gong et al. 2019 ; Wang et al. 2023a ) . We identify three barriers to practical WS adoption: 1) Implementation complexity: existing methods require architecture-specific adaptations and may not generalize to modern components ( Su et al. 2021 ; Gu and Dao 2023 ) ; 2) Unclear growth limits: practitioners lack guidance on how much a given checkpoint can be grown before WS becomes counterproductive ( Xie et al. 2022 ; Wang et al. 2023a ) ; 3) Hyperparameter opacity under scaling: the interaction between WS methods, hyperparameters, and model and data scaling budgets is under-explored, creating tuning overhead that undermines efficiency gains ( Fetterman et al. 2023 ; Dey et al. 2025 ) . Moreover, in scaling regimes where both models and data grow to improve final convergence loss, measuring relative early-stage convergence speed may not adequately capture practical requirements ( Shin et al. 2024 ; Unlu 2026 ) . In this work, we attempt to systematically investigate when WS is truly effective, in a first-principles manner, to address these barriers. We limit the discussion scope, both for clarity and computation, for the case when a base model checkpoint is available to be WS -ed to a larger model scale. We assume that we are aware of the dataset the base was trained on, and that some of its training hyperparameters may be available, but we treat it as a black-box when it comes to the training compute invested in it (unlike the assumption made by Liew and Kato 2025 ). We believe this better reflects a real-world scenario where only model checkpoints are publicly released. Our contributions are: 1. We generalize warmstarting growth into shrink-zero-perturb ( SZP ), a simple architecture-agnostic operation, and find that exact function-preservation is not required for effective warmstarting. 2. We empirically establish an upper bound on the growth ratio of a given checkpoint, such that, beyond this bound, training the grown model from scratch converges as fast as warmstarting it. 3. Using scaling-law fits, we show that for a given warmstarting method and base model size, the convergence advantage over training from scratch persists only within a bounded token budget, and we predict where this crossover occurs. The paper proceeds as follows: Section 2 reviews prior warmstarting methods and guidance; Section 3 introduces SZP and its design motivation; Section 4 reports hyperparameter transfer, growth limits, and scaling-law predictions; and Section 5 summarizes practical guidelines and limitations. We release our model and training pipelines in the linked repository 1 1 1 https://github.com/Neeratyoy/warmstarting_exps . 2 Related Works Model growth as initialization is the broad class of WS techniques that derive the weights of a larger model from a smaller, typically well-trained checkpoint, before beginning training. Here, the aim is to transfer the knowledge learned by the smaller model to improve the larger model’s training efficiency. Net2Net ( Chen et al. 2016 ) and Network Morphisms ( Wei et al. 2016 ) are among the earliest works exploring model growth for Deep Learning. At the time, the scope was limited to convolutional networks, but the core idea was carried forward to evolving architectures ( Gong et al. 2019 ; Chen et al. 2021 ; Yao et al. 2024 ; Samragh et al. 2024 ) . The benefit lies in being a one-time operation before training, requiring only weights from a smaller network of the same architectural family. A central idea in designing WS operations for initialization is that of function-preservation ( FP ), wherein the grown model’s output at the first forward pass matches that of the base model checkpoint used for WS . Introduced by Chen et al. 2016 for Deep Learning, FP has dominated the width-scaling

Read the complete text