Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
Kyle Moore, Jesse Roberts, Daryl Watson · 2025
Read Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models by Kyle Moore, Jesse Roberts, Daryl Watson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08204.txt.
Introduction
Kyle Moore, Jesse Roberts, and Daryl Watson test whether language models' self-reported uncertainty matches human uncertainty. Their paper appeared in 2025. Models that know when they are unsure can guide control systems and earn user trust. Most past work checked uncertainty against the model's own accuracy. This team compares many real-time uncertainty measures with how unsure groups of humans are on the same questions.
Several measures lined up strongly with human uncertainty, even though they did not match which answers humans prefer. The same measures also showed fair calibration against correctness, by correlation and by distribution checks. The authors say a model's hesitation carries real information about question difficulty. Designers can use it live, to decide when the model should answer, hedge, or hand off.
Catalog record
| CREATOR | Kyle Moore, Jesse Roberts, Daryl Watson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08204
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models Kyle Moore, Jesse Roberts, Daryl Watson Abstract There has been much recent interest in evaluating large language models for uncertainty calibration to facilitate model control and modulate user trust. Inference time uncertainty, which may provide a real-time signal to the model or external control modules, is particularly important for applying these concepts to improve LLM-user experience in practice. While many of the existing papers consider model calibration, comparatively little work has sought to evaluate how closely model uncertainty aligns to human uncertainty. In this work, we evaluate a collection of inference-time uncertainty measures, using both established metrics and novel variations, to determine how closely they align with both human group-level uncertainty and traditional notions of model calibration. We find that numerous measures show evidence of strong alignment to human uncertainty, even despite the lack of alignment to human answer preference. For those successful metrics, we find moderate to strong evidence of model calibration in terms of both correctness correlation and distributional analysis. Introduction Figure 1: LLMs and humans have similar uncertainty at inference time and frequently agree on overt selection. However, their cloze probability-based preference orderings are not aligned. A sizeable body of work has developed around the identification and quantification of uncertainty in the outputs of transformer-based large language models (LLMs). Accurate uncertainty quantification (UQ) is an essential element in predicting model hallucinations and maintaining user trust. In service of that, UQ research has largely focused on developing and utilizing uncertainty measurement methods that are well-calibrated to model accuracy. A well calibrated measure is one that predicts well the model’s likelihood of generating a valid answer to the given context. Contexts with high certainty should have a low likelihood of being incorrect and vice versa. A subset of UQ work focuses on measures that are able to be calculated at any time during generation, without additional auxiliary generations. This has often been referred to as inference-time uncertainty quantification. Inference-time measures are uniquely useful in that they can provide a constant signal to the user or to external control modules without significant added computation. Existing research has not considered whether the investigated UQ measures align with human uncertainty. So, while research has investigated measures with significant calibration, the reported values may not correspond with human uncertainty, making the meaning of the values difficult to parse for users. Simultaneously, a growing body of research has emerged that seeks to identify human-like behaviors in a variety of LLM tasks and contexts. This has included behaviors as varied as theory of mind ( Ullman 2023 ; Amirizaniani et al. 2024 ; Strachan et al. 2024 ) , strategic preferences ( Roberts, Moore, and Fisher 2024 ; Duan et al. 2024 ) , and framing effects ( Jumelet, Zuidema, and Sinclair 2024 ; Nguyen 2024 ) . This work seeks to synthesize these two research thrusts by identifying uncertainty measures that are simultaneously calibrated and aligned to human uncertainty behavior. In particular, we investigate whether any of the uncertainty measures vary consistently with human uncertainty on a per-question basis. Given the difficulty in reliably quantifying uncertainty for an individual human, we approximate this by comparing model measures against disagreement among groups of human survey respondents. By evaluating LLM uncertainty alignment as well as calibration in inference-time UQ measures, this paper identifies a set of UQ measures which may be effective and more intuitively interpretable by human users—enabling important advances in LLM-human interaction and prompting the further study of alignment in model signals beyond overt action. This paper specifically contributes to the existing understanding of LLM uncertainty by: N oting that top-p selection in LLM decoding is functionally equivalent to the Bayesian highest density credible set, drawing an important but previously un-noted connection between the fields and inspiring our investigation of top-p as a measure of uncertainty. O bserving that many entropy-based inference-time uncertainty measures have significant human uncertainty alignment despite moderate choice selection and no preference ordering alignment. D eveloping a novel ground-truth distributional calibration measure based on shift in the Jensen-Shannon distance metric to directly evaluate the impact of certainty on answer distribution. S howing that the aligned inference-time measures show evidence of calibration in terms of both correctness correlation and ground-truth distributional calibration. Prior Work Uncertainty Quantification in LLMs is a broad field with numerous notions of uncertainty depending on context, task, and available resources. These are covered in a variety of surveys including Liu et al. 2025 ; Shorinwa et al. 2025 ; He et al. 2025 . Many of the most successful methods in terms of calibration, like monte-carlo dropout ( Shelmanov et al. 2021 ; Roberts et al. 2024b ) , rely on multiple generation steps to quantify uncertainty and cannot be readily adapted to quantify per-token inference-time uncertainty levels. Existing work on inference-time uncertainty quantification typically relies on perplexity ( Mora-Cross and Calderon-Ramirez 2024 ; Margatina et al. 2023 ; Jiang et al. 2021 ) , maximum token probability ( Tian et al. 2023 ; Steyvers et al. 2025 ; Huang et al. 2025 ; Shrivastava, Liang, and Kumar 2023 ) , or entropy methods ( Kadavath et al. 2022 ; Huang et al. 2025 ) . Very few works have explicitly investigated the presence of human-like uncertainty responses in LLMs. This work was inspired by preliminary work by Moore et al. 2025 , which investigated human-similarity on a diverse set of uncertainty measures. Their work was limited in that the dataset consisted of less than 40 items and did not consider whether the measures which were aligned were also calibrated. We expand on those results by drastically expanding the dataset size and narrow our focus exclusively to inference-time measures. Our work is also novel in that it is the only extant work, to our knowledge, that simultaneously evaluates any uncertainty measures for both alignment and calibration. Other related work, including Argyle et al. 2023 ; Huang, Wu, and Wang 2025 , have used LLMs and uncertainty-aware procedures to simulate human group responses, but do not seek to establish human-like uncertainty measures. Inference-Time UQ This work is interested in investigating inference-time UQ methods, as inference-time calculation is necessary for model control mechanisms to have a signal to interpret and react. Non-inference-time methods are valuable tools for diagnostics, model comparison, etc., but are ill-suited for time-sensitive application. While the field of LLM UQ research is large and growing, it has focused on measures of uncertainty that have limited or no inference-time capabilities. These commonly include intuitive methods like self-reporting ( Zhou, Jurafsky, and Hashimoto 2023 ; Mielke et al. 2022 ; Band et al. 2024 ; Lin, Hilton, and Evans 2022 ; Tang, Shen, and Kejriwal 2024 ; Chaudhry, Thiagarajan, and Gorur 2024 ; Shrivastava, Liang, and Kumar 2023 ; Tian et al. 2023 ; Xiong et al. 2023 ; Belém et al. 2024 ) , multi-inference consistency ( Lin, Hilton, and Evans 2022 ; Kadavath et al. 2022 ; Chen and Mueller 2024 ; Manakul, Liusie, and Gales 2023 ; Zhang et al. 2024 ) , and ensemble variation ( Wang, Aitchison, and Rudolph 2023 ; Roberts et al. 2024b ; Gal and Ghahramani 2016 ; Fomicheva et