When Scale is Fixed: Revisiting Pre-training Indicators for LLM Fine-tuning Performance

Hansi Zeng, Kai Hui, Honglei Zhuang, Zhen Qin, Zhenrui Yue, Hamed Zamani, Dana Alon · 2025

Read When Scale is Fixed: Revisiting Pre-training Indicators for LLM Fine-tuning Performance by Hansi Zeng, Kai Hui, Honglei Zhuang, Zhen Qin, Zhenrui Yue, Hamed Zamani, Dana Alon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12491.txt.

Introduction

Hansi Zeng and co-authors ask whether signs seen during the first training of a language model can predict how well it will do after fine-tuning, in a paper from 2025. A common sign, called perplexity, tracks how surprised a model is by text. It works well in studies of model size, but its value at a fixed size was unclear.

The team built fifty variants of a billion-parameter model, each trained with different settings or data. All were then fine-tuned and tested on a range of tasks. The task was framed as picking which of two models would do better. Perplexity proved to be a poor guide. New signs drawn from the training stage cut the error rate of such picks by more than half. The authors show the signs help in real model choices.

Catalog record

CREATORHansi Zeng, Kai Hui, Honglei Zhuang, Zhen Qin, Zhenrui Yue, Hamed Zamani, Dana Alon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12491

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Scale is Fixed: Revisiting Pre-training Indicators for LLM Fine-tuning Performance Hansi Zeng, Kai Hui, Honglei Zhuang, Zhen Qin, Zhenrui Yue, Hamed Zamani, Dana Alon Abstract While pre-trained available metrics, such as perplexity, correlates well with model performance at scaling-laws studies, their predictive capacities at a fixed model size remains unclear, hindering effective model selection and development. To address this gap, we formulate the task of selecting pretraining checkpoints to maximize downstream fine-tuning performance as a pairwise classification problem: predicting which of two LLMs, differing in their pre-training, will perform better after supervised fine-tuning (SFT). We construct a dataset using 50 1B parameter LLM variants with systematically varied pre-training configurations, e.g., objectives or data, and evaluate them on diverse downstream tasks after SFT. We first conduct a study and demonstrate that the conventional perplexity is a misleading indicator. As such, we introduce novel unsupervised and supervised proxy metrics derived from pre-training that successfully reduce the relative performance prediction error rate by over 50%. Despite the inherent complexity of this task, we demonstrate the practical utility of our proposed proxies in specific scenarios, paving the way for more efficient design of pre-training schemes optimized for various downstream tasks. 1 Introduction Large Language Models (LLMs) ( Google et al., 2024 ; OpenAI, 2023 ; Chowdhery et al., 2023 ; Grattafiori et al., 2024 ) are central to contemporary NLP, powering systems like Chatbots and specialized assistants. They are typically employed via few-shot prompting or task-specific fine-tuning. Despite the accessibility afforded by prompting, fine-tuning on downstream tasks is often indispensable for optimal model performance, particularly within specific application domains or when utilizing private data ( Singhal et al., 2025 ; Lee et al., 2024b ; Lai et al., 2023 ) . While LLMs demonstrably improve on supervised fine-tuning (SFT) tasks with increasing scale ( Zhang et al., 2024 ; Isik et al., 2025 ) , the substantial costs associated with larger models strongly motivate performance optimization at a fixed size. These efforts often concentrate on refining pre-training elements, such as data compositions ( Shen et al., 2024 ; Penedo et al., 2024 ) or training objectives ( Raffel et al., 2020 ; Tay et al., 2023a ; Tay et al., 2023b ) . This context underscores a critical need: the ability to reliably forecast the post-SFT performance of same-size LLM variants using only indicators available during pre-training. Although metrics like perplexity correlate well with scaling-driven performance gains (lower perplexity generally corresponds to better few-shot ( Grattafiori et al., 2024 ) and fine-tuning ( Isik et al., 2025 ) results as model size expands), their predictive efficacy for fine-tuning outcomes within a constant model size remains uncertain. Practically, dependable predictors are essential to avoid the prohibitive expense of fine-tuning numerous checkpoints. This requirement is especially pronounced for monitoring and guiding decisions throughout the lengthy pre-training cycles (often months) of very large models ( Liu et al., 2024a ; Grattafiori et al., 2024 ) , and also when subsequent fine-tuning involves substantial datasets, including potentially stopping unpromising runs early. Figure 1: Mean pairwise error rates across three SFT tasks (separate plots). Each plot compares perplexity, the best individual proxy (Section 3 ), and the learning-to-compare proxy (shown on the x-axis). The y-axis represents the error rate, defined as the proportion of mis-classified LLM pairs regarding post-SFT performance. To investigate the predictability of fine-tuning outcome within feasible computational limits, our study employs a controlled methodology using smaller models. We train multiple variants of a 1B-parameter language model, each incorporating systematic variations in its pre-training configuration. We then evaluate the accuracy of potential predictors by comparing their values at the final pre-training checkpoints against the models’ eventual performance after supervised fine-tuning (SFT). While simplified, we posit that this approach provides representative insights into the core question–whether fine-tuning outcome can be reliably predicted during and after pre-training. Specifically, we generated 50 distinct 1B-parameter LLM variants by systematically altering pre-training objectives ( Raffel et al., 2020 ; Tay et al., 2023a ; Tay et al., 2023b ) , data composition strategies ( Shen et al., 2024 ) , and data processing techniques such as filtering and domain tagging ( Penedo et al., 2024 ) . These pre-trained models were subsequently fine-tuned across a diverse suite of tasks, including commonsense reasoning, retrieval-augmented generation and closed-book question answering. Specifically, we select five datasets ( Clark et al., 2019 ; Zellers et al., 2019 ; Bisk et al., 2019 ; Mihaylov et al., 2018 ; Sakaguchi et al., 2021 ) for commonsense reasoning, four ( Kwiatkowski et al., 2019 ; Joshi et al., 2017 ; Yang et al., 2018 ; Ho et al., 2020 ) for retrieval-augmented generation, and two ( Kwiatkowski et al., 2019 ; Joshi et al., 2017 ) for closed-book question answering. To align with the practical model development scenarios where the primary goal is to identify top performers from a set of candidate models, we formulate the prediction challenge as a pairwise classification task: given two pre-trained models differing only in pre-training, the goal is to predict which model will achieve superior performance after SFT. We begin by evaluating conventional perplexity, computed using a causal language modeling objective ( Brown et al., 2020 ) , as a predictor of SFT performance. Surprisingly, this standard metric correlates poorly with the downstream results of the LLMs after fine-tuning, resulting in prediction error rates exceeding 60% across all three evaluated tasks–worse than the 50% error rate of random guessing (Figure 1 ). Motivated by prior work ( Raffel et al., 2020 ; Tay et al., 2023a ; Von Oswald et al., 2023 ) , we then introduce alternative pre-training available proxies, including span corruption-based perplexity and k-shot learning performance ( Min et al., 2022 ) . These proxies yield substantially improved prediction accuracy; the best-performing proxy for each task reduces the error rate by nearly half compared to conventional perplexity (Figure 1 ). For example, in the commonsense reasoning task, the error rate drops from 69.4\% to 31.3\% . Furthermore, we propose a learning-to-compare (LTC) framework that integrates multiple proxies via supervised classification. By learning interactions across these heterogeneous signals, the LTC approach achieves more robust performance estimation and further decreases the predictive error. The contributions of this paper are three-folds. • We present the first formal study focused on predicting post-SFT performance across LLMs of identical size using pretraining signals—departing from prior scaling-based analyses. • Our work demonstrates the insufficiency of perplexity for this prediction task and introduces novel unsupervised and supervised proxies achieving over a 50% reduction in error rates. • Our work underscores the challenges of predicting supervised fine-tuning performance and confirms the practical value of the proposed proxies in specific scenarios; to foster further research, we provide the SFT performance data and individual pre-training proxy measurements in Appendix Table 6 . 2 Problem Definition and Setup This section defines the problem and details the setup, including the generation of diverse LLM variants, the target SFT tasks, and the pre-training signals used as prediction proxies. 2.1 LLM Variants and Target SFT Tasks LLM model variations. To approximate pre-training studies while maintaining reasonable computational resources, we continuously trained a 1B parameter LLM with 100B tokens, systematically ablating pre-training objectives,

Read the complete text