Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs

Beier Luo, Cheng Wang, Hongxin Wei, Sharon Li, Xuefeng Du · 2026

Read Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs by Beier Luo, Cheng Wang, Hongxin Wei, Sharon Li, Xuefeng Du (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04277.txt.

Introduction

Beier Luo and four co-authors tackle a side effect of post-training large language models in their 2026 arXiv paper. Models get better at tasks but worse at judging their own confidence, leaning toward systematic overconfidence. Earlier fixes matched a post-trained model's final output pattern to its pre-trained version, which the authors argue misses what changes during inference itself.

They separate two failure patterns: confidence drift, where final confidence swells while the underlying decision path stays much the same, and process drift, where that path itself wanders. Their method, Dual-Align, corrects both, matching final distributions and steadying the layers where paths split, by learning a single temperature value. Experiments are reported to cut calibration error toward a supervised benchmark while keeping post-training gains intact.

Catalog record

CREATORBeier Luo, Cheng Wang, Hongxin Wei, Sharon Li, Xuefeng Du
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs Beier Luo, Cheng Wang, Hongxin Wei, Sharon Li, Xuefeng Du Abstract Post-training improves large language models (LLMs) but often worsens confidence calibration, leading to systematic overconfidence. Recent unsupervised post-hoc methods for post-trained LMs (PoLMs) mitigate this by aligning PoLM confidence to that of well-calibrated pre-trained counterparts. However, framing calibration as static output-distribution matching overlooks the inference-time dynamics introduced by post-training. In particular, we show that calibration errors arise from two regimes: (i) confidence drift , where final confidence inflates despite largely consistent intermediate decision processes, and (ii) process drift , where intermediate inference pathways diverge. Guided by this diagnosis, we propose Dual-Align , an unsupervised post-hoc framework for dual alignment in confidence calibration. Dual-Align performs confidence alignment to correct confidence drift via final-distribution matching, and introduces process alignment to address process drift by locating the layer where trajectories diverge and realigning the stability of subsequent inference. This dual strategy learns a single temperature parameter that corrects both drift types without sacrificing post-training performance gains. Experiments show consistent improvements over baselines, reducing calibration errors and approaching a supervised oracle. Figure 1: Illustration of our method: DUAL-ALIGN. Our approach addresses both confidence drift and process drift. For confidence drift, we align the LLMs’ confidence using the objective \mathcal{L}_{\rm Conf} (Left). For process drift, we first identify the Peak Divergence Layer (PDL), then calculate the Inferential Stability Entropy (ISE) with respect to the process drift between the PLM and PoLM, and align it using the objective \mathcal{L}_{\rm Process} (Right). 1 Introduction Post-training methods such as instruction tuning and reinforcement learning from human feedback, substantially improves large language model (LLM) alignment and adaptability across tasks ( 38 ; 21 ; 47 ) . Yet it also introduces new challenges in uncertainty estimates, often amplifying over-confidence relative to the pre-trained language models (PLMs) ( 1 ; 30 ) . To circumvent this, researchers have explored post-hoc confidence calibration, such as temperature scaling (TS) ( 7 ) for post-trained LMs (PoLMs): aligning predicted probabilities with empirical accuracy so models behave cautiously under uncertainty ( 43 ) . Recent unsupervised methods, such as DACA ( 22 ) , use the confidence of the well-calibrated PLM on unlabled data as a reference to calibrate the PoLM. To avoid potential conflicts from new knowledge introduced by post-training, DACA chooses to only align on samples where predictions are consistent between PLM and PoLM. However, this selective alignment strategy is inherently data-inefficient, as it discards all samples where the models disagree. More critically, by focusing solely on matching the final output confidence, it treats calibration as a static, surface-level matching problem. This fails to address the complex drifts in the model’s intermediate inference process induced by post-training, which are often the root cause of miscalibration. We raise a key question here: How does post-training alter the decision process of LLMs, and can we use that understanding to calibrate them more effectively? To answer this, we begin by investigating the different behavioral regimes of the PLM and PoLM by analyzing their differences w.r.t. the layer-wise predictions and final outputs. Our analysis at Figure 2 reveals two distinct post-training phenomena: (i) In samples where the PoLM and PLM agree on the final answer, their intermediate decision processes are largely consistent, yet the PoLM’s final confidence is systematically inflated—a phenomenon we term confidence drift . (ii) Conversely, in samples where they disagree, the models’ decision pathways diverge sharply at a specific intermediate layer, causing their inference trajectories to split and lead to different answers. We term this more fundamental change process drift . These observations motivate a calibration approach that addresses both phenomena at their source. Our contributions. To this end, we propose Dual-Align , a novel post-hoc LLM calibration framework (Figure 1 ) that treats calibration as a dual alignment problem. It performs (1) confidence alignment to correct surface-level overconfidence by matching the PoLM’s final-layer output distribution with the PLM’s. Our motivation for pursuing deeper alignment in the models’ inference process arises from a key observation: post-training creates a problematic pattern in which extreme overconfidence is coupled with unnaturally low Inferential Stability Entropy (ISE) (Figure 3 ), calculated over the LLM inference trajectory through different layers. To rectify this, we introduce a novel (2) process alignment , which first identifies the Peak Divergence Layer (PDL)—the point at which the inference pathways of the PLM and PoLM models diverge most significantly—and then aligns the PoLM’s ISE with the PLM’s healthier distribution from that layer forward. Importantly, our framework interpolates between these two objectives on a per-sample basis using a divergence-derived weight coefficient. This approach produces a temperature parameter that adapts to different miscalibration regimes, while preserving the performance gains achieved through post-training . Both theoretical results (Proposition 1 ) and empirical findings (Section 5 ) demonstrate that Dual-Align achieves substantial improvements, reducing the Expected Calibration Error (ECE) by more than 30% across various advanced LLM architectures compared to strong baselines. 2 Preliminaries Probability distribution across transformer layers. Formally, we define the input prompt as a sequence of tokens \bm{x}=\{x_{1},x_{2},\dots,x_{N}\} and our analysis focuses on the final token, x_{N} , as its hidden state is used to generate the model’s prediction. At each layer l\in[1,L] of a transformer model ( 36 ) , the hidden state for this token is conceptually updated as: \bm{h}^{l}(x_{N})=\bm{h}^{l-1}(x_{N})+\text{Attn}^{l}(x_{N})+\text{MLP}^{l}(x_{N}), (1) where \bm{h}^{l}\in\mathbb{R}^{d_{\text{model}}} denotes the hidden state at the l -th layer. Using LogitLens ( 26 ) , we can project any intermediate hidden state \bm{h}^{l}(x_{N}) into the vocabulary space via the unembedding matrix W_{U}\in\mathbb{R}^{V\times d_{\text{model}}} , with V as the vocabulary size. Since the embedding \bm{h}^{l}(x_{N}) encapsulates information from the entire input \bm{x} , we denote the resulting per-layer logits as \bm{z}^{l}(\bm{x})=W_{U}\cdot\bm{h}^{l}(x_{N})\in\mathbb{R}^{V}. (2) Our analysis primarily focuses on Multiple-Choice Question Answering (MCQA) problems, which typically present a set of options, such as \mathcal{Y}=\{A,B,C,D\} . The probability of each option at layer l is given by p_{i}^{l}(\bm{x})=\frac{\exp(z^{l}_{i}(\bm{x}))}{\sum_{j\in\mathcal{Y}}\exp(z^{l}_{j}(\bm{x}))},\quad i\in\mathcal{Y}. (3) Confidence calibration for PoLMs. We aim to calibrate a post-trained language model PoLM, denoted by f , using a pre-trained language model PLM, g , as a reference. In the context of a multiple-choice question, the model’s prediction, \hat{y}_{f}(\bm{x}) , is the choice with the highest probability at the final-layer L , and this maximum probability value is taken as its confidence, \hat{P}(\bm{x})=\underset{i\in\mathcal{Y}}{\max}\,p_{i}^{L}(\bm{x}) . A model is considered perfectly calibrated if its confidence matches its true accuracy, i.e., \Pr\big(Y=\hat{y}\,\big|\,\hat{P}=\beta\big)=\beta , where Y is the ground-truth. A standard metric to measure this discrepancy is the Expected Calibration Error (ECE) ( 24 ) . In practice, ECE is estimated empirically by partitioning K samples into M bins {b_{1},b_{2},\dots,b_{M}} based on the model’s predicted confidence scores, and then computed as: \mathrm{ECE}=\sum_{m=1}^{M}\frac{|b_{m}|}{K}\,\big|\mathrm{acc}(b_{m})-\mathrm{conf}(b_{m})\big|, (4) where \mathrm{acc}(b_{m}) and \mathrm{conf}(b_{m}) are the average accuracy and confidence in bin b_{m} . A smaller ECE indicates better calibration performance of the model. While PLMs are often well-calibrated, literature recognize that post-training often degrades this property, leading to overconfident predictions ( 39 ; 22 ; 15 ) , as shown in Figure 5 . Post-hoc calibration methods. Post-hoc calibration adjusts a model’s confidence without altering its predictions. A popular supervised method is Temperature Scaling (TS) ( 7 ) , which softens

Read the complete text