Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution’s Characteristics
Lorenzo Jaime Yu Flores 1,2 Ori Ernst 1,2 Jackie Chi Kit Cheung 1,2,3 1 Mila - Quebec AI Institute 2 McGill University 3 Canada CIFAR AI Chair, Mila {lorenzo.flores, ori.ernst, cheungja}@mila.quebec · 2025
Read Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution’s Characteristics by Lorenzo Jaime Yu Flores 1,2 Ori Ernst 1,2 Jackie Chi Kit Cheung 1,2,3 1 Mila - Quebec AI Institute 2 McGill University 3 Canada CIFAR AI Chair, Mila {lorenzo.flores, ori.ernst, cheungja}@mila.quebec (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00637.txt.
Introduction
Lorenzo Jaime Yu Flores and colleagues improve confidence calibration in text generation. Their 2025 arXiv paper tackles a practical trust problem. Users could review low-confidence outputs to catch bad or risky text. That plan fails if confidence scores do not match real accuracy. Generation is tricky because many answers can all be valid.
A confident model may split probability across several good answers. Older metrics read that split as doubt and misjudge the model. The team proposes metrics based only on the output distribution. No extra training or hand-built rules are needed. Tests used BART and Flan-T5 on summary, translation, and question tasks. Calibration improved across those tasks. Better scores could make human review of AI text far more useful.
Catalog record
| CREATOR | Lorenzo Jaime Yu Flores 1,2 Ori Ernst 1,2 Jackie Chi Kit Cheung 1,2,3 1 Mila - Quebec AI Institute 2 McGill University 3 Canada CIFAR AI Chair, Mila {lorenzo.flores, ori.ernst, cheungja}@mila.quebec |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00637
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution’s Characteristics Lorenzo Jaime Yu Flores 1,2 Ori Ernst 1,2 Jackie Chi Kit Cheung 1,2,3 1 Mila - Quebec AI Institute 2 McGill University 3 Canada CIFAR AI Chair, Mila {lorenzo.flores, ori.ernst, cheungja}@mila.quebec Abstract Well-calibrated model confidence scores can improve the usefulness of text generation models. For example, users can be prompted to review predictions with low confidence scores, to prevent models from returning bad or potentially dangerous predictions. However, confidence metrics are not always well calibrated in text generation. One reason is that in generation, there can be many valid answers, which previous methods do not always account for. Hence, a confident model could distribute its output probability among multiple sequences because they are all valid. We propose task-agnostic confidence metrics suited to generation, which rely solely on the probabilities associated with the model outputs without the need for further fine-tuning or heuristics. Using these, we are able to improve the calibration of BART and Flan-T5 on summarization, translation, and QA datasets. Our work can be found at https://github.com/ljyflores/calibrated-confidence-for-nlg 1 Introduction Confidence scores are scores derived from a model’s output, which are interpreted as the model’s estimation of its own output’s quality. These scores can be used in real-world applications to flag uncertain predictions in automated decision-making systems ( Malinin and Gales 2021 ) , which could prompt further human review ( Xiao et al. 2020 ) , or force the model to abstain from answering when unsure ( Liu et al. 2020 ; Kamath et al. 2020 ) . To be useful, we want these scores to correlate with the output’s quality. A common approach to estimating confidence is through probability-based methods, which rely on the probabilities assigned by the model to output tokens. Most existing methods focus on the sequence with the highest probability, which we refer to as the top sequence ( Murray and Chiang 2018 ; Zablotskaia et al. 2023 ; Huang et al. 2023 ; Zhao et al. 2020 ; Perlitz et al. 2023 ; Malinin and Gales 2021 ) . A high probability for the top sequence suggests strong confidence in a particular prediction, while a lower value indicates uncertainty. This approach is effective for tasks with a single correct answer. However, it faces significant challenges when applied to tasks with multiple valid outputs, as in many generation tasks. In such cases, a low top probability may not reflect a lack of confidence but rather that the model has identified several valid sequences (See Figure 1 ). Ideally, for open-ended tasks, a confident model would distribute high probabilities across multiple good sequences while assigning lower probabilities to less suitable options, while in classification, confidence can be indicated by a single high top probability. To address this limitation, we propose new probability-based confidence estimation methods that consider the probabilities of multiple sequences instead of focusing solely on the top one. We introduce two methods: the first calculates the probability ratio between the highest-ranked sequences and the rest, while the second evaluates the thinness of the distribution’s tail. Our experiments demonstrate that these metrics outperform existing baselines across three open-ended text generation tasks: translation, QA, and summarization. Figure 1: We illustrate the difference in interpretation of confidence in classification vs generation. Suppose a model generated output probability distributions for four different inputs; each bar is the prob. assigned to one class/sequence. In classification, only the 1st output would show model confidence, as it assigned most probability to one class. In generation, the first 3 outputs could show confidence because multiple sequences were valid. 2 Related Work Probability-Based Methods These methods rely on the model outputs to compute token-level probabilities or entropy ( Murray and Chiang 2018 ; Zablotskaia et al. 2023 ; Zhao et al. 2020 ; Perlitz et al. 2023 ; Kumar and Sarawagi 2019 ; Huang et al. 2023 ; Malinin and Gales 2021 ) . Other work uses natural language inference models to group similar sequences before computing entropy ( Lin et al. 2023 ; Kuhn et al. 2023 ; Nikitin et al. 2024 ) . Similarity/Disagreement Based Methods When answers can be sampled from models (e.g., through dropout), self-consistency can be used to measure confidence: consistency across the top answers indicates confidence while variance indicates uncertainty ( Xiao et al. 2020 ; Schmidt et al. 2022 ; Lakshminarayanan et al. 2017 ) . Fine-Tuning Based Methods In addition, other methods also fine-tune additional models to predict the correctness or confidence of the output ( Yaldiz et al. 2024 ; Kamath et al. 2020 ; Malinin et al. 2019 ; Fathullah et al. 2023 ) . Out of Distribution Detection (OOD) Methods OOD can also be used to detect if a sample is in the training distribution, in which case a model is assumed to be confident ( Liu et al. 2020 ; Vazhentsev et al. 2023 ) . Verbalized Confidence Scores With the increased conversational ability of LLMs, recent work prompts the model to give a confidence score with its answer ( Lin et al. 2022 ; Tian et al. 2023 ; Kapoor et al. 2024 ; Han et al. 2024 ) . Our work is closest to the probability-based methods; they are easily adaptable and task agnostic. They do not require metrics or NLI models to measure similarity, computation for OOD detection, or models that can verbalize their confidence. 3 Method Problem Definition We define confidence as a score computed using the model outputs, that describes its assessment of its prediction quality. We want to compute the model’s sample-level confidence for its output, that is positively correlated (i.e. calibrated) to the output’s quality, measured by the evaluation metric used for the task (e.g. automated metrics, human evaluation). Formally, \displaystyle\text{Confidence}(x,\hat{y},\phi)\propto\text{Quality}(y,\hat{y}), where x is the input, y is the target, \hat{y} is the prediction, and \phi are the model parameters. At inference time, we run beam search to generate N sequences. Each sequence’s probability is obtained by taking the product of the individual token probabilities. Given the i-th beam \hat{y}^{(i)} : p_{\hat{y}^{(i)}}(x)=\prod_{t}p(\hat{y}_{t}^{(i)}|\hat{y}_{<t}^{(i)},x) Methods We account for the fact that there can be multiple valid outputs by measuring two characteristics that we hypothesize are present in all confident outputs regardless of the number of valid sequences (See Figure 2 ). The first characteristic of a confident model is that it distinguishes good from average or bad sequences, and subsequently assigns higher probability to a select set of sequences it deems as good compared to other sequences. Figure 2: We hypothesize that a confident model’s output would have a steep slope and long tail; colors added for illustration purposes only. Ratio This motivates the ratio method: we measure how much more confident the model is in one of its best beams p_{\hat{y}^{(1)}} , versus one of its average beams p_{\hat{y}^{(k)}} , where p_{\hat{y}^{(1)}} to p_{\hat{y}^{(k)}} are sorted in descending order. This captures the intuition that a confident model will assign more probability to its best sequence than to an average sequence, whereas an unconfident model would assign similar probabilities to them. We expect the optimal value of k to differ by task, as different tasks can have different levels of diversity in