Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning
Lorenzo Jaime Yu Flores 1,2 Cesare Spinoso di-Piano 1,2 Jackie Chi Kit Cheung 1,2,3 1 Mila - Quebec AI Institute 2 McGill University 3 Canada CIFAR AI Chair, Mila {lorenzo.flores,cheungja}@mila.quebec · 2026
Read Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning by Lorenzo Jaime Yu Flores 1,2 Cesare Spinoso di-Piano 1,2 Jackie Chi Kit Cheung 1,2,3 1 Mila - Quebec AI Institute 2 McGill University 3 Canada CIFAR AI Chair, Mila {lorenzo.flores,cheungja}@mila.quebec (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08974.txt.
Introduction
Flores, Spinoso di-Piano, and Cheung ask whether confidence scores for language model text still mean what they should after supervised fine-tuning. That routine step adapts a model to a new task. Across translation, question answering, and mathematical reasoning, they run 216 configurations. Calibration worsens in 112 cases and improves in 104, shifting with closeness to the training data rather than answer quality alone. Because the direction cannot be predicted from the original model, off-the-shelf confidence scores lose practical value for spotting right answers after tuning.
Catalog record
| CREATOR | Lorenzo Jaime Yu Flores 1,2 Cesare Spinoso di-Piano 1,2 Jackie Chi Kit Cheung 1,2,3 1 Mila - Quebec AI Institute 2 McGill University 3 Canada CIFAR AI Chair, Mila {lorenzo.flores,cheungja}@mila.quebec |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58766 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08974
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning Lorenzo Jaime Yu Flores 1,2 Cesare Spinoso di-Piano 1,2 Jackie Chi Kit Cheung 1,2,3 1 Mila - Quebec AI Institute 2 McGill University 3 Canada CIFAR AI Chair, Mila {lorenzo.flores,cheungja}@mila.quebec Abstract Uncertainty quantification is a set of techniques that measure confidence in language models. They can be used, for example, to detect hallucinations or alert users to review uncertain predictions. To be useful, these confidence scores must be correlated with the quality of the output. However, recent work found that fine-tuning can affect the correlation between confidence scores and quality. Hence, we investigate the underlying behavior of confidence scores to understand its sensitivity to supervised fine-tuning (SFT). We find that post-SFT, the correlation of various confidence scores degrades, which can stem from changes in confidence scores due to factors other than the output quality, such as the output’s similarity to the training distribution. We demonstrate via a case study how failing to address this miscorrelation reduces the usefulness of the confidence scores on a downstream task. Our findings show how confidence metrics cannot be used off-the-shelf without testing, and motivate the need for developing metrics which are more robust to fine-tuning. Our code is available at https://github.com/ljyflores/sensitivity-of-calibration-to-sft.git . 1 Introduction Uncertainty quantification (UQ) is a set of techniques for measuring the confidence of language models, which has increasingly been applied towards generation tasks. These confidence scores can be used to detect hallucinations ( Wang et al. 2024 ; Manakul et al. 2023 ) , select answers with higher quality ( Wang et al. 2023 ) , self-evaluate model outputs ( Ren et al. 2023 ) , and prompt users to review uncertain predictions ( Xiao et al. 2020 ; Malinin and Gales 2021 ; Liu et al. 2020 ; Kamath et al. 2020 ) (e.g. ‘‘Outputs with average token probability 1 1 1 Average token probability is an example of a confidence score used in generation tasks ( Murray and Chiang 2018 ; Zablotskaia et al. 2023 ) from 0.20–0.25 often have low BLEU scores (range: 5–10); review before proceeding”). To be useful in these applications, the scores generated by UQ techniques (i.e. confidence scores), must be correlated with the quality of the output. Hence, an important question is how correlated these metrics are with quality, especially when used off-the-shelf with models fine-tuned in different ways. This is important as various UQ metrics and packages are designed to allow any white-box model to be plugged in ( Shelmanov et al. 2025 ) . However, recent work showed how fine-tuning can make models overconfident ( Rathi et al. 2025 ; Leng et al. 2025 ) , thus making confidence scores miscorrelated with quality. This challenges whether these UQ metrics retain their correlation to quality when used with any fine-tuned model. Hence, we investigate the sensitivity of the correlation between confidence scores and quality to SFT. We study probability-based and self-consistency based confidence scores which only rely on the model’s token-level probabilities or outputs. These are assumed to approximate models’ output quality since language models’ output probabilities are trained to maximize the expected utility of one output ( Wang and Holmes 2024 ) . In our paper, we first fine-tune models on NLG tasks, and find that the correlation of various scores changes significantly after SFT, and varies with the number of samples and epochs used. Across 144 task-model-UQ metric configurations, correlation degrades in a third of the cases post-SFT (48/144), showing that these metrics need to be thoroughly checked before deployment. Figure 1: Before SFT, a model had a relatively low confidence score in a correct answer, whereas post-SFT, it had a much higher confidence in an incorrect answer, demonstrating a case of relative overconfidence Because correlation relies on sample-level confidence and evaluation scores being aligned, we study sample-level changes in these scores to understand how miscorrelation occurs post-SFT and to motivate interventions. We find that SFT is prone to making probability-based scores overconfident (see Figure 1 ) and self-consistency-based scores underconfident. Hence, the way in which miscorrelation occurs is not always the same, and different interventions may be needed for each metric. We also find that confidence scores are affected by factors beyond the output’s quality such as the output’s similarity to the training distribution, which must be accounted for to isolate the relationship with quality. To illustrate the consequences of miscorrelation on a downstream task, we use confidence scores to predict whether the output is correct on a QA task, and find that these confidence scores’ ability to identify correct answers decreases in 47% cases post-SFT. Thus, the usefulness of these metrics can suffer if miscorrelation is not addressed. To summarize, we ask: RQ1: How sensitive is the correlation of confidence metrics to different SFT parameters? RQ2: What are the underlying dynamics of confidence scores that negatively impact correlation with quality? We find that the correlation of various confidence metrics change drastically post-SFT, with different types of miscorrelation occurring, stemming from factors beyond the quality of the output. This can have negative impacts on downstream applications of UQ metrics. Hence, confidence metrics cannot be used off-the-shelf without testing. We argue for the need for developing metrics which are more robust to model finetuning, or at least, better reporting of the consistency of such metrics in future studies. 2 Related Work UQ Metrics Uncertainty quantification (UQ) encompasses a broad set of methods for identifying a model’s confidence in its answers. These include fine-tuning additional models to estimate model confidence ( Yaldiz et al. 2024 ; Kamath et al. 2020 ; Malinin et al. 2019 ; Fathullah et al. 2023 ) , or detecting out-of-distribution (OOD) samples, for which a model’s confidence is assumed to be lower ( Liu et al. 2020 ; Vazhentsev et al. 2023 ) . Other work studies having models verbalize a confidence score ( Lin et al. 2022a ; Tian et al. 2023 ; Kapoor et al. 2024 ; Han et al. 2024 ) , which relies on larger language models which are capable of doing so. In this study, we focus on two types of methods: Probability-based methods use the model’s output token probabilities to compute confidence ( Murray and Chiang 2018 ; Zablotskaia et al. 2023 ; Zhao et al. 2020 ; Perlitz et al. 2023 ; Kumar and Sarawagi 2019 ; Huang et al. 2023 ; Malinin and Gales 2021 ; Flores et al. 2025 ) . These can be augmented by other models to take into account the similarity of model’s outputs ( Lin et al. 2023 ; Kuhn et al. 2023 ; Nikitin et al. 2024 ) . Self-consistency methods obtain multiple answers from models (e.g., through dropout, beam search), and measure how similar (i.e. self-consistent) the model’s predictions are: self-consistency across the top answers indicates confidence while variance indicates uncertainty ( Xiao et al. 2020 ; Schmidt et al. 2022 ; Lakshminarayanan et al. 2017 ) . Recently, methods combined probability and self-consistency methods into one metric ( Vashurin et al. 2025 ) . We study these methods as they can be used with any language model, and do not require additional fine-tuning or specialized modules.