Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning

Kazuki Yano, Shun Kiyono, Sosuke Kobayashi, Sho Takase, Jun Suzuki · 2026

Read Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning by Kazuki Yano, Shun Kiyono, Sosuke Kobayashi, Sho Takase, Jun Suzuki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16127.txt.

Introduction

Yano and co-authors skip learning rate decay in pretraining to gain fine-tuning strength. Decay schedules chase the lowest pretraining loss by habit. Their Warmup-Stable-Only plan holds the rate flat after warmup. Tests span 1B and 8B parameter models in size. Decay wins right after pretraining on raw scores. After supervised fine-tuning, the flat-rate models win instead. Loss maps show decay traps models in sharper minima. Flat minima adapt better to later training stages. The pattern holds through mid-training and over-training runs.

Catalog record

CREATORKazuki Yano, Shun Kiyono, Sosuke Kobayashi, Sho Takase, Jun Suzuki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59880
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16127

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning Kazuki Yano, Shun Kiyono, Sosuke Kobayashi, Sho Takase, Jun Suzuki Abstract We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised fine-tuning (SFT). Decay-based learning rate schedulers are widely used to minimize pre-training loss. However, despite their widespread use, how these schedulers affect performance after SFT remains underexplored. In this paper, we examine Warmup-Stable-Only (WSO), which maintains a constant learning rate after warmup without any decay. Through experiments with 1B and 8B parameter models, we show that WSO consistently outperforms decay-based schedulers in terms of performance after SFT, even though decay-based schedulers may exhibit better performance after pre-training. The result also holds across different regimes with mid-training and over-training. Loss landscape analysis further reveals that decay-based schedulers lead models into sharper minima, whereas WSO preserves flatter minima that support adaptability. These findings indicate that applying LR decay to improve pre-training metrics may compromise downstream adaptability. Our work also provides practical guidance for training and model release strategies, highlighting that pre-training models with WSO enhances their adaptability for downstream tasks. 1 Introduction Learning rate (LR) scheduling is arguably one of the most critical yet operationally challenging aspects of large language model (LLM) pre-training. Although Cosine decay has been conventionally employed in numerous models ( Brown et al. 2020 ; Le et al. 2022 ; Touvron et al. 2023a ) , it has proven inflexible in recent training paradigms such as continual pre-training, as it requires heuristic tuning of the LR from the decayed value ( Hägele et al. 2024 ; Ibrahim et al. 2024 ) . To address this inflexibility, recent studies have introduced Warmup-Stable-Decay (WSD), which keeps the LR constant through most of pre-training and decays it only briefly at the end ( Hu et al. 2024 ; Liu et al. 2024a ; Wen et al. 2025 ) . These previous studies, regardless of the details of the design choices, decayed the LRs to optimize the performance of pre-trained models. However, the more critical factor for real applications is the performance after post-training, such as supervised fine-tuning (SFT). Drawing on the findings of Sun & Dredze 2025 and Springer et al. 2025 , which show that a strong pre-training model does not necessarily imply superior performance after SFT, it is questionable to schedule LRs to the decayed value based on pre-training performance. In this study, we provide a comprehensive empirical investigation of LR schedulers during pre-training in terms of performance after SFT. In particular, we examine an underestimated scheduling, Warmup-Stable-Only (WSO), which removes the decay phase from WSD and maintains constant LR to the end. We show that WSO consistently achieves superior performance after SFT compared to decay-based schedulers, through experiments on 1B and 8B models (Figure 1 ). Furthermore, we demonstrate that WSO is also effective under modern training paradigms, including mid-training ( OLMo et al. 2024 ; Meta 2024c ) and over-training ( Sardana et al. 2024 ; Gadre et al. 2025 ) . To understand why WSO yields superior SFT performance, we draw on insights from the transfer learning literature ( Ju et al. 2022 ; Liu et al. 2023 ) , which suggest that models in flatter regions of the loss landscape tend to exhibit better adaptability. Through an analysis of sharpness values, we show that models trained with WSO reside in flatter regions than those trained with other decay-based LR schedulers, and are therefore more adaptable to post-training tasks. Our contributions are as follows: (1) We provide the systematic demonstration that WSO consistently outperforms decay-based schedulers on downstream tasks after SFT, with comprehensive evidence across 1B and 8B models and diverse evaluation benchmarks. (2) We show that WSO similarly benefits mid-training and over-training scenarios, achieving superior SFT performance compared to conventional decay-based schedulers. (3) We reveal through loss landscape analysis that WSO preserves flatter minima than decay-based schedulers, explaining why models trained with WSO achieve better performance after SFT. Figure 1: Learning rate schedulers used in pre-training and their impact on performance after supervised fine-tuning (SFT). Warmup-Stable-Only (WSO), which removes the decay phase, achieves the highest performance after SFT. 2 Preliminaries Recent LLMs are typically built with a staged training scheme. The most common and fundamental training pipeline consists of two stages, namely pre-training and post-training. In this section, we describe these training stages and review the LR schedulers commonly employed during pre-training. Pre-training. Pre-training forms the foundation of LLM development, where models learn general language understanding from massive text corpora by minimizing the next-token prediction loss. Recently, pre-training has sometimes consisted of multiple stages: standard pre-training and mid-training ( OLMo et al. 2024 ) . We describe mid-training in detail later (Section 2.2 ), and conduct experiments with both the standard pre-training and the multi-stage setup. Post-training. Post-training adapts pre-trained models to target tasks, enabling them to follow human instructions and avoid generating harmful outputs. Post-training includes techniques such as supervised fine-tuning (SFT), preference tuning (e.g., DPO ( Rafailov et al. 2023 ) ), and RL-based alignment ( Ouyang et al. 2022 ) . While post-training could be a multi-stage process with many design choices still under active exploration, SFT is relatively standardized and serves a core stage. In this paper, we focus on SFT as the canonical post-training stage and evaluate the performance after SFT 1 1 1 The computational cost of pre-training is typically much larger than that of other stages, so identifying a better pre-training configuration has a substantial impact on the efficiency of LLM construction. In this study, we focus on evaluating LR schedulers during large-scale pre-training and characterize the potential of non-decay schedulers based on the performance after SFT. An exploration of complex combinations of LR scheduling spanning multiple post-training stages is left to future work. . 2.1 Task Definition Practically, LLM developers evaluate models at multiple stages, selecting the best-performing one as the starting point for the subsequent stage. We define \mathtt{Task}_{\rm s}(M) as a function that, for a given LLM M , returns the performance on a set of pre-defined tasks used to assess the target stage s , where s\in\{\rm pre,post\} denotes the training stage, with \rm pre indicating pre-training and \rm post indicating post-training, respectively. We write M_{2}[M_{1}] to denote the model M_{2} trained with some configuration and initialization with M_{1} , where M_{\rm rand} indicates a model whose weights are randomly initialized. Moreover, we introduce \mathcal{M}_{\rm pre} and \mathcal{M}_{\rm post} to represent the sets of models obtained through pre-training and post-training, respectively, with various hyperparameter configurations. A typical training pipeline for building LLMs can therefore be expressed as follows: \displaystyle\widehat{M}_{\rm pre} \displaystyle=\displaystyle\argmax_{M_{\rm{pre}}\in\mathcal{M}_{\rm pre}}\left\{\mathtt{Task}_{\rm pre}(M_{\rm{pre}}[M_{\rm rand}])\right\}, (1) \displaystyle\widehat{M}_{\rm post} \displaystyle=\displaystyle\argmax_{M_{\rm{post}}\in\mathcal{M}_{\rm{post}}}\left\{\mathtt{Task}_{\rm post}(M_{\rm{post}}[\widehat{M}_{\rm pre}[{M}_{\rm rand}]])\right\}. This formulation may lead to a suboptimal solution in terms of the performance of the final model, namely, \widehat{M}_{\rm post} , since selecting the best-performing models at intermediate stages does not guarantee achieving the best performance in the end. Therefore, conceptually, we would like to consider the following search problem to obtain a better final model for this training pipeline: \displaystyle\widehat{M}_{\rm post}=\argmax_{(M_{\rm{pre}},M_{\rm{post}})\in(\mathcal{M}_{\rm{pre}},\mathcal{M}_{\rm{post}})}\ \left\{\mathtt{Task}_{\rm post}(M_{\rm{post}}[M_{\rm{pre}}[M_{\rm rand}]])\right\}. (2) The primary objective of this paper is to empirically examine the search

Read the complete text