On Predicting the Post-training Potential of Pre-trained LLMs
Xiaoyuan Li, Yubo Ma, Kexin Yang, Moxin Li, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu · 2026
Read On Predicting the Post-training Potential of Pre-trained LLMs by Xiaoyuan Li, Yubo Ma, Kexin Yang, Moxin Li, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11978.txt.
Introduction
Xiaoyuan Li and seven colleagues predict a base language model's post-training promise before post-training begins. Standard benchmarks such as MMLU poorly reflect how models adapt in open-ended tasks, leading to wasteful model choice. Their RuDE system sidesteps the output gap through response sorting, building matched paired contrasts via fine-grained rubric breaks under a 4C scheme. Tests showed over 90 percent match with post-training scores, and RL checks showed it spots high-potential smaller models.
Catalog record
| CREATOR | Xiaoyuan Li, Yubo Ma, Kexin Yang, Moxin Li, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 116254 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11978
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Predicting the Post-training Potential of Pre-trained LLMs Xiaoyuan Li, Yubo Ma, Kexin Yang, Moxin Li, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu Abstract The performance of Large Language Models (LLMs) on downstream tasks is fundamentally constrained by the capabilities acquired during pre-training. However, traditional benchmarks like MMLU often fail to reflect a base model’s plasticity in complex open-ended scenarios, leading to inefficient model selection. We address this by introducing a new task of predicting post-training potential - forecasting a base model’s performance before post-training. We propose RuDE ( Ru bric-based D iscriminative E valuation), a unified framework that bypasses the generation gap of base models by leveraging response discrimination. Guided by our systematic 4C Taxonomy , RuDE constructs controlled contrastive pairs across diverse domains by fine-grained rubric violations. Extensive experiments demonstrate a correlation greater than 90% with post-training performance. Crucially, validation via Reinforcement Learning (RL) confirms that RuDE effectively identifies high-potential smaller models that outperform larger counterparts, offering a compute-efficient mechanism for foundation model development. 1 Introduction The development of Large Language Models (LLMs) typically follows a two-stage pipeline: Pre-training followed by Post-training ( 37 ; 30 ; 19 ) . Pre-training over large text corpora allows the LLM to capture linguistic patterns and store extensive knowledge 24 . Subsequently, post-training enables the LLM to follow human instructions 26 , align with human preferences 21 , and perform complex reasoning tasks 28 , unlocking the underlying capabilities captured during pre-training. Recently, studies show that the performance of post-trained largely depends on the capabilities acquired during pre-training stage 39 ; 10 ; 44 . Therefore, selecting an optimal pre-trained model is crucial for effective post-training and for saving computational resources. However, a critical research gap remains in how to pre-evaluate the potential of pre-trained models for effective post-training. Existing evaluations typically measure pre-trained model quality using accuracy on multiple-choice knowledge benchmarks such as MMLU ( 13 ) or using perplexity metrics ( 41 ) . These metrics, however, are insufficient to capture the model’s post-training potential —its expected performance on post-training tasks that are typically open-ended and require both complex instruction following ability and understanding of human intents. We observe that high performance of pre-trained models on multiple-choice knowledge benchmarks correlates weakly with their post-training performance on open-ended instruction following tasks ( cf. Figure 1 ). This motivates us to investigate the following problem. How can we evaluate the relative post-training potential of pre-trained models across diverse tasks? We identify the following challenges in addressing this problem. (1) Generation Gap : Pre-trained models are optimized for next-token prediction and generally lack the instruction-following capability to respond to open-ended queries ( 44 ) , making direct evaluation challenging. (2) Unified Objective Evaluation : Unlike multiple-choice questions, which can be evaluated via exact answer matching, open-ended responses involve diverse criteria such as formatting, helpfulness, and factual correctness. A generalized evaluation framework is therefore needed to quantify these objectives across diverse tasks ( 18 ) . To address the first challenge, inspired by the Generation-Evaluation Consistency hypothesis ( 20 ) , we estimate a pre-trained model’s ability to generate desired open-ended responses based on its capability to discriminate between preferred and dispreferred responses. To tackle the second challenge, we introduce the 4C Taxonomy , providing a systematic framework for evaluating open-ended generation across four distinct aspects: Competence , Content , Control , and Compliance . Post-training tasks are evaluated using discriminative response sets constructed from task-specific combinations of these aspects, allowing the framework to adapt to diverse tasks. To this end, we propose RuDE ( Ru bric-based D iscriminative E valuation) a unified framework for evaluating the relative post-training potential of pre-trained models across diverse tasks. We begin by manually defining fine-grained rubrics, i.e., evaluation criteria, for each aspect of the 4C Taxonomy . For a comprehensive evaluation, we adapt four open-ended datasets into the response discrimination format, covering diverse domains: Medical, Legal & Finance, Complex Instruction Following, and Creative Writing , and define rubric sets for each task. We propose a generator-verifier pipeline to automatically construct discriminative response sets, where preferred responses satisfy all rubrics and dispreferred responses violate some. Evaluation results on pre-trained models show a strong Pearson correlation ( >90\% ) with their fully instruction-tuned performance. Experiments on Reinforcement Learning (RL) further validate our approach, showing that a smaller pre-trained model (Qwen3-4B-Base) predicted to have higher potential indeed outperforms a larger counterpart (Qwen2.5-7B-Base) after post-training. Our contributions are summarized as follows: • We introduce the new task of evaluating the post-training potential of pre-trained LLMs. • We propose a unified evaluation framework for post-training potential across diverse tasks via rubric-based response discrimination based on our 4C Taxonomy. • We perform extensive experiments on instruction tuning and RL across multiple models, demonstrating the strong effectiveness of RuDE . 2 Task Formulation In this section, we formalize the task of evaluating post-training potential. We first define the problem as a ranking task over pre-trained models (§ 2.1 ). We then introduce discriminative evaluation as a proxy to address the generation gap (§ 2.2 ). Finally, we state our core hypothesis connecting discriminative ability to predicting post-training potential (§ 2.3 ). 2.1 Problem Definition The performance of post-trained models is largely determined by their pre-training foundation, yet no reliable method exists to predict which pre-trained model will perform best after post-training. We formalize this as a ranking problem. Consider a set of pre-trained base models \Theta=\{\theta_{1},\dots,\theta_{n}\} and a target post-training task T . Let \texttt{PostTrain}(\theta,T) denote the model obtained by applying a post-training algorithm ( e.g., RL) to base model \theta on task T , and let S_{\text{gen}}(\texttt{PostTrain}(\theta,T)) be the resulting generative performance score on open-ended evaluation. Our goal is to find a scoring function S(\theta,T) , computable on the base model alone, such that the ranking induced by S over \Theta is consistent with the ranking induced by S_{\text{gen}} after post-training: S(\theta_{i},T)>S(\theta_{j},T)\;\Longrightarrow\;S_{\text{gen}}(\texttt{PostTrain}(\theta_{i},T))>S_{\text{gen}}(\texttt{PostTrain}(\theta_{j},T)) (1) A reliable scoring function S would allow practitioners to select the most promising base model without expensive trial-and-error post-training. As shown in Figure 1 , existing proxies for base model quality—such as accuracy on multiple-choice benchmarks like MMLU—correlate weakly with post-training performance on open-ended tasks, because they measure knowledge recall rather than the model’s capacity to produce aligned, high-quality responses. This motivates the search for a more predictive scoring function. Figure 1 : Correlation Analysis: Traditional Benchmarks vs. Post-Training Performance. The X-axis represents the accuracy of base models on widely used knowledge benchmarks and the Y-axis denotes the corresponding models’ post-training performance on HealthBench after instruction tuning. The Pearson correlation coefficients ( corr ) between them indicate a weak correlation, demonstrating that traditional benchmarks can not predict domain-specific post-training potential. 2.2 Discriminative Evaluation as a Proxy The first challenge identified in the introduction is the generation gap : pre-trained models, optimized for next-token prediction, generally cannot follow instructions or produce well-structured open-ended responses ( 44 ) . This makes it impractical to directly evaluate their generative quality on post-training tasks. To address this, inspired by the Generation-Evaluation Consistency hypothesis ( 20 ) , we shift from evaluating what a model can generate to evaluating what it can discriminate . The key insight is that a model’s ability to