Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator

Beier Luo, Shuoyuan Wang, Sharon Li, Hongxin Wei · 2025

Read Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator by Beier Luo, Shuoyuan Wang, Sharon Li, Hongxin Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16690.txt.

Introduction

Tuned language models grow overconfident, and a 2025 paper by Beier Luo and co-authors recalibrates them without labeled data. Base models judge their own odds well, but tuning for human taste breaks that balance. Per-task labels for fixing confidence are scarce, which blocks standard fixes.

Their method watches where the base and tuned models disagree. On those items, the base model's caution carries useful signal about true accuracy. A single temperature knob is then set using only agreement items, avoiding the under-confidence trap. Theory in the paper explains why naive use of all items fails. The method promises safer confidence scores for tuned models in high-stakes uses.

Catalog record

CREATORBeier Luo, Shuoyuan Wang, Sharon Li, Hongxin Wei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16690

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator Beier Luo, Shuoyuan Wang, Sharon Li, Hongxin Wei Abstract Post-training of large language models is essential for adapting pre-trained language models (PLMs) to align with human preferences and downstream tasks. While PLMs typically exhibit well-calibrated confidence, post-trained language models (PoLMs) often suffer from over-confidence, assigning high confidence to both correct and incorrect outputs, which can undermine reliability in critical applications. A major obstacle in calibrating PoLMs is the scarcity of labeled data for individual downstream tasks. To address this, we propose Disagreement-Aware Confidence Alignment (DACA), a novel unsupervised method to optimize the parameters (e.g., temperature τ \tau ) in post-hoc confidence calibration. Our method is motivated by the under-confidence issue caused by prediction disagreement between the PLM and PoLM while aligning their confidence via temperature scaling. Theoretically, the PLM’s confidence underestimates PoLM’s prediction accuracy on disagreement examples, causing a larger τ \tau and producing under-confident predictions. DACA mitigates this by selectively using only agreement examples for calibration, effectively decoupling the influence of disagreement. In this manner, our method avoids an overly large τ \tau in temperature scaling caused by disagreement examples, improving calibration performance. Extensive experiments demonstrate the effectiveness of our method, improving the average ECE of open-sourced and API-based LLMs (e.g., GPT-4o) by up to 15.08 % \% on common benchmarks. 1 Introduction Post-training has been a critical procedure to ensure large language models (LLMs) generate helpful, honest, and harmless responses [ Weng et al., 2023 , Kumar et al., 2025 ] . While post-trained language models (PoLMs) perform well on various downstream tasks [ Achiam et al., 2023 , DeepSeek-AI and et al., 2025 ] , their reliability and trustworthiness still remain an open challenge. In principle, a reliable LLM should not only demonstrate high confidence in its correct generations but also exercise caution in uncertain situations [ Thirunavukarasu et al., 2023 , Dahl et al., 2024 ] . Previous studies [ Achiam et al., 2023 , Zhu et al., 2023 ] show that post-training, especially RLHF [ Christiano et al., 2017 , Stiennon et al., 2020 ] , compromises the well-calibrated confidence estimation of pre-trained language models (PLMs), resulting in over-confidence issues of PoLMs. This gives rise to the importance of confidence calibration for PoLMs, ensuring the confidence score associated with the generation should reflect its ground truth correctness likelihood. Compared to expensive training methods, post-hoc calibration methods such as temperature scaling [ Guo et al., 2017 ] are more practical for LLMs due to their high efficiency [ Shen et al., 2024 , Xie et al., 2024 ] . However, a primary challenge of post-hoc calibration methods is their dependence on labeled data. In practice, generating a reliable labeled dataset for tasks such as mathematics problem solving and medical diagnosis is particularly challenging and time-consuming due to the high level of domain expertise required. Such difficulty is further compounded by the fact that temperature scaling cannot perform effectively given limited labeled data [ Mozafari et al., 2018 , Liang et al., 2020 ] . In contrast, unlabeled data is ubiquitous in real-world deployment scenarios and easy to collect without requiring human intervention. This creates an underutilized resource: vast amounts of unlabeled data are already available during LLM operation, yet are not leveraged for calibration. Thus, this paper studies an unexplored and practical perspective: How can we achieve effective confidence calibration for PoLMs using unlabeled data in an unsupervised manner? To calibrate PoLMs without relying on labeled data, we introduce Disagreement-Aware Confidence Alignment ( DACA )—a simple and effective post-hoc method that leverages the well-calibrated confidence scores of PLMs. A natural starting point of our idea is to align the confidence of PoLMs with that of PLMs on an unlabeled validation set, minimizing the divergence between the predictive distributions of the PLM and PoLM over all samples. However, we find that this direct confidence alignment can paradoxically lead to under-confidence in the PoLM—when the two models disagree on a prediction, the PLM’s confidence often underestimates the actual correctness of the PoLM’s output. Our theoretical analysis reveals that such prediction disagreement can drive the optimization to increase the temperature parameter excessively, further exacerbating the under-confidence issue. Motivated by our theory, DACA mitigates this issue by decoupling the influence of disagreement examples from the confidence alignment process. Specifically, it optimizes the temperature parameter using only agreement examples—those where the PLM and PoLM make identical predictions. This ensures that confidence alignment occurs only when the PLM’s scores are a trustworthy proxy for correctness. As a result, DACA yields more conservative and reliable temperature estimates, avoiding the calibration failures of naive alignment (see Figure 2(b) ). Extensive experiments with both open-sourced and API-based LLMs on common benchmarks demonstrate the effectiveness of the DACA method for confidence calibration. Notably, DACA achieves performance comparable to labeled temperature scaling, even in the absence of labeled data. For example, DACA improves the average Expected Calibration Error (ECE) of the Gemma-3-12B-Instruct model [ Team et al., 2025 ] across 57 subjects of the MMLU dataset [ Hendrycks et al., 2021b ] , reducing it from 23.68 \% to 8.60 \% . In comparison, TS only reduces the ECE to 9.75 \% . Importantly, DACA is applicable even in scenarios where post-trained and pre-trained models differ in architecture, making it more efficient for the calibration of large-scale PoLMs. For instance, DACA reduces the ECE of GPT-4o [ Hurst et al., 2024 ] from 21.23 \% to 6.99 \% when calibrated using the pre-trained Gemma-3-12B model on the MedMCQA dataset [ Pal et al., 2022 ] . Furthermore, our method can be applied to open-ended question-answering tasks and offers benefits for selective classification. Codes are publicly available at https://github.com/ml-stat-Sustech/Disagreement-Aware-Calibration . We summarize our contributions as follows. 1. We show that the well-calibrated outputs of PLMs on unlabeled data can be leveraged to calibrate PoLMs. Theoretically, we demonstrate that prediction disagreement can impair calibration performance when directly aligning the confidence of PLMs and PoLMs. 2. Our proposed post-hoc method DACA, formalizes the confidence calibration problem by harnessing the target-specific unlabeled data in the wild. This formulation offers strong practicality and flexibility for real-world applications. 3. We empirically show that DACA enhances the calibration of both open-sourced and API-based PoLMs across various datasets. Moreover, our method applies to open-ended QA tasks and enhances selective classification. 2 Preliminaries 2.1 Confidence Calibration for LLMs In this work, we focus on the confidence calibration problem of question answering for the Post-trained Language Model (PoLM), denoted as f . Our method primarily targets Multiple-Choice QA (MCQA) and can be extended to open-ended QA. For MCQA with choices \mathcal{Y}=\{A,B,C,D\} and prompt \bm{x} , let z_{f}(\bm{x})\in\mathbb{R}^{|\mathcal{Y}|} be the logits of f . The predicted probabilities and confidence are \displaystyle p_{f}(y=j\mid\bm{x})=\frac{\exp\big(z_{f,j}(\bm{x})\big)}{\sum_{j^{\prime}\in\mathcal{Y}}\exp\big(z_{f,j^{\prime}}(\bm{x})\big)},\qquad\hat{P}(\bm{x})=\max_{j\in\mathcal{Y}}\,p_{f}(y=j\mid\bm{x}), (1) with prediction \hat{Y}(\bm{x})=\arg\max_{j}p_{f}(y=j\mid\bm{x}) . We denote the n -th prompt in a dataset by \bm{x}_{n}=x_{n,t_{n}},...,x_{n,t_{2}},x_{n,t_{1}} , which is a sequence of t_{n} tokens, with its corresponding response denoted as y_{n} . Formally, given a prompt \bm{x} , a perfectly calibrated model satisfies, \Pr(Y=\hat{Y}\mid\hat{P}=\beta)=\beta,\quad\forall\beta\in[0,1], (2) where \hat{Y}=\arg\max_{y}p(y|\bm{x}) is the predicted response, and \hat{P}=\max_{y}p(y|\bm{x}) is the corresponding confidence score [ Guo et al., 2017 ] . To quantify the degree of miscalibration, expected calibration error (ECE) [ Naeini

Read the complete text