Reasoning about Uncertainty: Do Reasoning Models Know When They Don’t Know?
Zhiting Mei, Christina Zhang, Tenny Yin, Justin Lidard, Ola Shorinwa, Anirudha Majumdar · 2025
Read Reasoning about Uncertainty: Do Reasoning Models Know When They Don’t Know? by Zhiting Mei, Christina Zhang, Tenny Yin, Justin Lidard, Ola Shorinwa, Anirudha Majumdar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18183.txt.
Introduction
Zhiting Mei and co-authors present an AI safety paper, published in 2025, asking whether reasoning language models know when they are unsure. Such models solve hard problems step by step, yet they also produce confident wrong answers, a risk when people rely on them. The team measures calibration, the match between stated confidence and real accuracy, across leading reasoning models and benchmarks.
Three findings stand out. The models were typically overconfident, above all when wrong. Deeper reasoning made the overconfidence worse, not better. Asking a model to reason about its own chain of thought, a method the authors call introspective uncertainty checks, improved calibration. The work gives deployers a clearer picture of when a reasoning model's confidence deserves trust.
Catalog record
| CREATOR | Zhiting Mei, Christina Zhang, Tenny Yin, Justin Lidard, Ola Shorinwa, Anirudha Majumdar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning about Uncertainty: Do Reasoning Models Know When They Don’t Know? Zhiting Mei, Christina Zhang, Tenny Yin, Justin Lidard, Ola Shorinwa, Anirudha Majumdar Abstract Reasoning language models have set state-of-the-art (SOTA) records on many challenging benchmarks, enabled by multi-step reasoning induced using reinforcement learning. However, like previous language models, reasoning models are prone to generating confident, plausible responses that are incorrect (hallucinations). Knowing when and how much to trust these models is critical to the safe deployment of reasoning models in real-world applications. To this end, we explore uncertainty quantification of reasoning models in this work. Specifically, we ask three fundamental questions: First, are reasoning models well-calibrated? Second, does deeper reasoning improve model calibration? Finally, inspired by humans’ innate ability to double-check their thought processes to verify the validity of their answers and their confidence, we ask: can reasoning models improve their calibration by explicitly reasoning about their chain-of-thought traces? We introduce introspective uncertainty quantification (UQ) to explore this direction. In extensive evaluations on SOTA reasoning models across a broad range of benchmarks, we find that reasoning models: (i) are typically overconfident, with self-verbalized confidence estimates often greater than 85 % 85\% particularly for incorrect responses, (ii) become even more overconfident with deeper reasoning, and (iii) can become better calibrated through introspection (e.g., o3-Mini and DeepSeek R1) but not uniformly (e.g., Claude 3.7 Sonnet becomes more poorly calibrated). Lastly, we conclude with important research directions to design necessary UQ benchmarks and improve the calibration of reasoning models. 1]Princeton University \contribution [*]Equal contribution. keywords Reasoning about Uncertainty, Reasoning Models, LLMs, Uncertainty Quantification. Figure 1: To examine if reasoning models know when they don’t know, we ask three fundamental questions: (i) are reasoning models well-calibrated? (ii) does deeper reasoning improve model calibration? and (iii) can introspection improve calibration? We find that reasoning models are typically overconfident and become more overconfident with deeper reasoning. However, through introspection, reasoning models can provide better calibrated confidence estimates. 1 Introduction Recent breakthroughs in large language models (LLMs) and vision-language models (VLMs) have largely been driven by reasoning language models—language models that are trained to perform multi-step reasoning through reinforcement learning, e.g., Claude 3.7 [ 2 ] , o3-Mini [ 37 ] , Gemini 2 Flash Thinking [ 44 ] , and DeepSeek R1 [ 18 ] . By breaking down complex tasks into subtasks and analyzing and synthesizing responses from the resulting subtasks, reasoning models have achieved new state-of-the-art (SOTA) performance on a broad range of challenging real-world tasks, such as programming, e.g., Codeforces [ 34 ] ; math, e.g., AIME [ 3 ] ; engineering and science, e.g., GPQA [ 38 ] ; and arts, e.g., MMLU [ 21 ] , outperforming domain experts on many benchmarks. Despite their remarkable capabilities, SOTA reasoning language models often generate confident but factually-incorrect responses to questions, known as hallucinations [ 5 , 31 ] , similar to their non-reasoning counterparts. Consequently, the safe integration of reasoning models into real-world tasks requires knowing when and how much to trust these models, i.e., uncertainty quantification (UQ) of these models. Prior work has examined UQ of non-reasoning models, showing that non-reasoning models are poorly calibrated in general [ 52 ] . However, UQ of reasoning models remains an unexplored frontier. To address this knowledge gap, we examine uncertainty quantification of reasoning models to better evaluate their trustworthiness. Specifically, in this work, we ask three critical questions (illustrated in Figure 1 ). First, are reasoning models well-calibrated? We evaluate the calibration of reasoning models across a broad range of benchmarks to examine underconfidence/overconfidence in these models across different problem domains. Our findings reveal that reasoning models are typically overconfident, often expressing confidence estimates greater than 85\% (in increments of 5\% ) even when they are incorrect. Given that overconfidence is often masked by high accuracy, we find that reasoning models are well-calibrated in knowledge-retrieval tasks and poorly calibrated in tasks requiring domain expertise, e.g., graduate-level research domains. Moreover, our results highlight that prompt engineering is significantly less effective with reasoning models, unlike non-reasoning models. Second, does deeper reasoning improve model calibration? Prior work [ 37 ] has shown that reasoning models can achieve higher accuracy through inference-time scaling of reasoning effort, measured by the number of available reasoning tokens. Here, we explore the effects of deeper reasoning on model calibration across low, medium, and high amounts of reasoning. Our evaluations reveal that reasoning models become more overconfident with deeper reasoning. Specifically on wrongly-answered questions, reasoning models become significantly more overconfident as we scale the reasoning effort. Lastly, humans often double-check their problem-solving approach to verify their answers and estimate their confidence in the validity of the answers [ 22 ] . In line with this observation, we ask the question: can reasoning models improve their calibration by explicitly reasoning about their chain-of-thought traces? We introduce introspective uncertainty quantification to explore this question. Introspective UQ utilizes a multi-stage reasoning paradigm to estimate the confidence in the accuracy of a given response: a model first reasons about a given question, and then a second model reasons about the chain of thought and final answer produced by the first model in order to assess its confidence in the answer. We find that more critical introspection improves the calibration of reasoning models, e.g., o3-Mini and DeepSeek R1, although the calibration of Claude 3.7 Sonnet worsens. 2 Related Work Reasoning in LLMs. Prior work [ 48 , 28 , 47 ] has demonstrated that language models can achieve significantly higher accuracy by generating intermediate reasoning steps, known as a chain of thought (CoT). Chain-of-thought prompting enables remarkable zero-shot or few-shot learning in LLMs, enabling generalist LLMs to outperform task-specific, finetuned models [ 48 ] . However, the complexity of the few-shot CoT examples significantly influences the induced accuracy of the models [ 12 ] , spurring research on prompt engineering for effective chain-of-thought reasoning [ 55 , 23 , 20 ] . Subsequent work introduces multi-path reasoning strategies to enable backtracking on incorrect reasoning paths, e.g., Tree-of-Thought prompting [ 53 ] and Graph-of-Thoughts prompting [ 7 ] which constructs parallel reasoning traces; least-to-most prompting [ 56 ] , which solves a series of subproblems in sequence; and reasoning via planning [ 19 ] , which performs Monte Carlo Tree Search on a reasoning tree to find the optimal reasoning path. However, existing approaches require domain expertise to construct effective chain-of-thought prompts/examples, posing a challenge. By training language models to reason using reinforcement learning (RL), recent work [ 25 , 18 , 2 , 37 , 44 , 45 ] has shown that reasoning language models can solve complex tasks more accurately than non-reasoning LLMs. Through inference-time scaling, reasoning models can be made to think deeper to break down complex problems into multiple steps, achieving SOTA performance on many benchmarks [ 2 , 37 , 44 , 18 ] . However, the effects of reasoning on the calibration of confidence estimates of these models remains unknown, motivating uncertainty quantification of reasoning models. Uncertainty Quantification of LLMs. Uncertainty quantification of LLMs has gained notable prominence given the tendency of LLMs to hallucinate. In general, existing UQ methods for LLMs eschew traditional techniques such as Bayesian networks [ 26 ]