Reasoning Models Better Express Their Confidence

Dongkeun Yoon, Seungone Kim, Sohee Yang, Sunkyoung Kim, Soyeon Kim, Yongil Kim, Eunbi Choi, Yireun Kim, Minjoon Seo · 2025

Read Reasoning Models Better Express Their Confidence by Dongkeun Yoon, Seungone Kim, Sohee Yang, Sunkyoung Kim, Soyeon Kim, Yongil Kim, Eunbi Choi, Yireun Kim, Minjoon Seo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14489.txt.

Introduction

Dongkeun Yoon and co-authors show that reasoning models express their confidence better than standard language models. Their 2025 paper deals with a reliability problem: models often fail to say how sure they are.

Six reasoning models were benchmarked against their non-reasoning counterparts on six datasets. The reasoning models calibrated their confidence better in nearly every setting. The authors trace the gain to slow thinking behaviors: weighing alternatives and backtracking mid-chain. Those habits let a model adjust its confidence as it reasons, so the final number tracks its true chance of being right. Better calibration helps users decide when to trust an answer.

Catalog record

CREATORDongkeun Yoon, Seungone Kim, Sohee Yang, Sunkyoung Kim, Soyeon Kim, Yongil Kim, Eunbi Choi, Yireun Kim, Minjoon Seo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14489

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reasoning Models Better Express Their Confidence Dongkeun Yoon, Seungone Kim, Sohee Yang, Sunkyoung Kim, Soyeon Kim, Yongil Kim, Eunbi Choi, Yireun Kim, Minjoon Seo Abstract Despite their strengths, large language models (LLMs) often fail to communicate their confidence accurately, making it difficult to assess when they might be wrong and limiting their reliability. In this work, we demonstrate that reasoning models that engage in extended chain-of-thought (CoT) reasoning exhibit superior performance not only in problem-solving but also in accurately expressing their confidence. Specifically, we benchmark six reasoning models across six datasets and find that they achieve strictly better confidence calibration than their non-reasoning counterparts in 33 out of the 36 settings. Our detailed analysis reveals that these gains in calibration stem from the slow thinking behaviors of reasoning models (e.g., exploring alternative approaches and backtracking) which enable them to adjust their confidence dynamically throughout their CoT, making it progressively more accurate. In particular, we find that reasoning models become increasingly better calibrated as their CoT unfolds, a trend not observed in non-reasoning models. Moreover, removing slow thinking behaviors from the CoT leads to a significant drop in calibration. Lastly, we show that non-reasoning models also demonstrate enhanced calibration when simply guided to slow think via in-context learning, fully isolating slow thinking as the source of the calibration gains. 1 1 1 Our code is available at https://github.com/MattYoon/reasoning-models-confidence Question Who’s the Athenian reformer that set up the council of 500? Model CoT … Hmm, I remember that Athens had several important reformers, like Solon, Cleisthenes, and Pericles. … So maybe it wasn’t Solon. Then there’s Cleisthenes. … (1) Oh, and he also created the council of 500, which was … (2) Pericles came later, and he was … (3) but I don’t think he was the one who established the council of 500. … (4) So putting it all together, Cleisthenes is the reformer who … I don’t think I’m mixing up any details here. … Figure 1: R1-Distill-Qwen-32B dynamically refines its confidence throughout CoT ( left ) as it engages in various slow thinking behaviors ( right ). We collect the model’s answer and confidence at each token position by appending “ </think>\nAnswer: ” to terminate the reasoning process early. Note that the model’s answer is consistent and correct (“Cleisthenes”) at all points, while the confidence fluctuates. For visual clarity, the data was smoothed using a Butterworth low-pass filter. See Appendix A.4 for the full untruncated CoT. 1 Introduction A persistent weakness of large language models (LLMs) is their tendency to sound confident even when they are wrong ( 13 , 45 ) . This overconfidence threatens their reliability, especially in high-stakes scenarios ( 32 , 49 ) . Meanwhile, recent reasoning models like OpenAI’s o1 ( 30 ) and Deepseek-R1 ( 4 ) have demonstrated strong problem-solving capabilities through chain-of-thought (CoT) ( 43 ) eliciting slow thinking behaviors such as exploring alternative approaches and verifying their answers ( 24 , 6 ) . Yet, it remains underexplored whether such slow thinking behaviors also help LLMs to “know what they know” ( 16 ) and accurately communicate the limits of their knowledge. To this end, we present an extensive study of reasoning models’ ability to accurately estimate and express confidence in their output; a process known as verbalized confidence estimation ( 40 , 7 ) . We demonstrate that reasoning models exhibit superior confidence calibration compared to non-reasoning models, and that this improvement stems from slow thinking behaviors, which allow LLMs to dynamically adjust their confidence throughout the reasoning process. For instance, as shown in Figure 1 , the model’s confidence increases when it first verifies its answer (“Oh, and he also created the council of 500”), and decreases when it considers an alternative (“Pericles came later”). The confidence then rises again as it rejects the alternative (“but I don’t think he was”), and reaches its highest level with further verification (“I don’t think I’m mixing up any details”). We begin by benchmarking six reasoning models derived from four backbone LLMs against their non-reasoning, instruction-tuned counterparts across two knowledge-focused datasets (TriviaQA and NonambigQA) ( 15 , 23 , 17 ) and four reasoning-intensive datasets (subsets of SuperGPQA and MMLU-Pro) ( 42 , 21 ) . In 33 out of 36 settings, reasoning models strictly outperform their non-reasoning counterparts across all measured calibration metrics. Notably, reasoning models achieve better calibration even on knowledge-focused datasets, despite having comparable task accuracy to non-reasoning models. This highlights that the calibration gains are not simply derived from superior task performance. We then conduct a detailed analysis to support our claim that the calibration gains in reasoning models stem from slow thinking, which allows them to dynamically adjust and correct their confidence throughout the reasoning process. (1) We measure changes in calibration metrics throughout the CoT process and observe a steady, gradual improvement for reasoning models, with statistically significant trends ( p<0.05 ). Non-reasoning models show no significant trend and in some cases, surprisingly, even exhibit a worsening pattern. (2) In our ablation study, we find that the non-linear structure of slow thinking [ 6 ] , such as the model’s ability to explore alternatives and revise its reasoning, plays a critical role in improving calibration. Conversely, prompting the model to explicitly reason about its own confidence yields only limited gains in calibration. (3) Lastly, we observe that non-reasoning models can also benefit from slow thinking when simply guided via in-context learning ( 5 ) , supporting our claim that these improvements arise from the slow thinking process itself, rather than from some other factors inherent to reasoning models. 2 Related Work Confidence in LLMs Confidence refers to the model’s estimated probability that its answer is correct, while calibration measures how well this confidence aligns with actual correctness ( 8 ) . In other words, a well-calibrated model’s confidence is a reliable indicator of prediction correctness ( 27 ) . Therefore, assessing an LLM’s confidence and enhancing calibration enables more reliable and trustworthy models ( 7 ) , supporting a wide range of applications such as hallucination detection ( 22 , 41 ) , ambiguity detection ( 12 ) , uncertainty-guided data retrieval ( 14 ) , and uncertainty-aware LLM agents ( 10 ) . Various approaches have been proposed for confidence estimation , which is the process of inferring an LLM’s confidence in its predictions ( 7 ) . This includes measuring token-level probabilities of answers ( 9 ) , training proxy probe models on internal hidden states ( 16 ) , and applying supervised fine-tuning with labeled confidence data ( 46 ) . Nonetheless, these methods assume access to a model’s internal states or weights, which are usually unavailable in proprietary LLMs ( 36 , 44 ) . Additionally, they are often model-specific, necessitating readjustment or retraining when the underlying LLM changes ( 47 ) . An alternative approach that avoids these limitations involves sampling multiple responses for the same prompt and deriving confidence based on response consistency ( 22 ) . While this method is compatible with proprietary LLMs and is model-agnostic, its primary limitation is the increased computational cost due to repeated inference ( 47 ) . In this work, we focus

Read the complete text