CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought

Boxuan Zhang, Ruqi Zhang · 2025

Read CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought by Boxuan Zhang, Ruqi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17214.txt.

Introduction

Boxuan Zhang and Ruqi Zhang tackle trust in large language models in a 2025 paper. Such models write well but judge their own doubt poorly, which makes false claims hard to spot. Past doubt tools graded the prompt, not each reply, and often needed many sampled replies at high cost. Worse, models that show their steps tend to sound far too sure.

Their frame, called CoT-UQ, folds the model's own step-by-step thought into the doubt score for a single reply. It pulls key words from each step to gauge how firm the chain is. Readers get a cheaper, per-reply way to flag shaky answers.

Catalog record

CREATORBoxuan Zhang, Ruqi Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17214

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought Boxuan Zhang, Ruqi Zhang Abstract Large language models (LLMs) excel in many tasks but struggle to accurately quantify uncertainty in their generated responses. This limitation makes it challenging to detect misinformation and ensure reliable decision-making. Existing uncertainty quantification (UQ) methods for LLMs are primarily prompt-wise rather than response-wise, often requiring multiple response samples, which incurs high computational costs. Moreover, LLMs have been shown to be overconfident, particularly when using reasoning steps to derive their answers. In this work, we propose CoT-UQ , a response-wise UQ framework that integrates LLMs’ inherent reasoning capabilities through Chain-of-Thought (CoT) into the UQ process. CoT-UQ captures critical information during inference by extracting keywords from each reasoning step and assessing their importance to the final answer. This key reasoning information is then aggregated to produce a final uncertainty estimate. We conduct extensive experiments based on Llama Family with model sizes varying from 8B to 13B across logical and mathematical reasoning tasks. Experimental results demonstrate that CoT-UQ significantly outperforms existing UQ methods, achieving an average improvement of 5.9% AUROC compared to current UQ methods. The code is available at: https://github.com/ZBox1005/CoT-UQ 1 Introduction Large language models (LLMs) have demonstrated groundbreaking capabilities across a variety of applications ( Ouyang et al. 2022 ; Chowdhery et al. 2023 ; OpenAI 2024b ; DeepSeek-AI 2025 ) . Particularly, prompting techniques like Chain-of-Thought (CoT) ( Wei et al. 2022 ) have significantly enhanced LLMs reasoning capabilities, ranging from multi-round conversation ( Long 2023 ; Chen et al. 2023 ) , logical reasoning ( Creswell et al. 2022 ; Duan et al. 2024b ) and mathematical reasoning ( Yao et al. 2024 ; Shao et al. 2024 ) . However, LLMs often unpredictably hallucinate ( Manakul et al. 2023 ) , i.e., making plausible but incorrect statements ( Ji et al. 2023 ) , limiting their deployment in safety-critical applications ( Clusmann et al. 2023 ) . Figure 1: Comparison of existing UQ strategies with ours. Directly estimating the uncertainty of a generated incorrect answer leads to overconfidence, which is exacerbated by using CoT to derive the answer. We tackle this challenge by integrating CoT into the UQ process with keywords extraction and importance scores. To improve the reliability of LLMs, uncertainty quantification (UQ) has emerged as a key strategy for determining when humans can trust LLM-generated outputs. However, existing UQ methods for LLMs are primarily prompt-wise ( Malinin and Gales 2021 ; Kuhn et al. 2023 ; Ling et al. 2024 ) . That is, uncertainty is calculated at the prompt level rather than for each individual response. These methods require multiple response samples per prompt, leading to high computational costs and inference time. Besides, some studies ( Kadavath et al. 2022 ; Miao et al. 2023 ) propose leveraging an LLM’s own ability to evaluate the uncertainty of its responses without relying on external knowledge. However, these approaches suffer from overconfidence issues, particularly when reasoning steps, such as Chain-of-Thought (CoT), are used before deriving the final answer ( Fu et al. 2025 ) . Overconfidence has been attributed to the model’s inherent bias toward trusting its own outputs ( Mielke et al. 2022 ; Lin et al. 2022 ) . To enable response-wise reliable UQ, we propose utilizing LLMs’ powerful reasoning capabilities in the UQ process. Our motivation comes from an intuitive insight: providing access to the reasoning path allows the model to use additional context for confidence calibration, leading to a more informed assessment of the final answer’s uncertainty. As illustrated in Figure 1 , while LLMs tend to be overconfident in their generated answers, and using CoT can further amplify this issue, we hypothesize that incorporating key reasoning information into the UQ process can effectively mitigate inflated confidence scores, resulting in better-calibrated uncertainty estimates. This naturally raises a critical research question: How can LLMs’ reasoning steps be utilized to improve uncertainty estimation? To answer this question, we propose a new framework, namely, C hain- o f- T hought enhanced U ncertainty Q uantification (CoT-UQ). At a high level, CoT-UQ provides response-wise uncertainty by integrating reasoning steps from CoT into the UQ process. Specifically, as illustrated in Figure 2 , CoT-UQ leverages the LLM’s reasoning process to extract keywords from each inference step and assess their importance in determining the final answer. By incorporating this critical information, CoT-UQ achieves better-calibrated uncertainty estimation, either by aggregating token probabilities of extracted keywords or by integrating the reasoning path into the self-evaluation process. We conducted extensive experiments to verify the effectiveness of our proposed framework. Under extensive evaluations, CoT-UQ achieves superior performance compared with different baselines, which reveals that LLMs have the potential to use their own reasoning to better express the trustworthiness of their generations. We also conduct a range of ablation studies and provide in-depth discussions via case studies. Our contributions can be summarized as the following: • Conceptually, we introduce a novel perspective to quantify uncertainty for LLMs by leveraging LLM’s internal reasoning knowledge. • Technically, we propose a new UQ framework, namely, C hain- o f- T hought enhanced U ncertainty Q uantification (CoT-UQ), which integrates the reasoning knowledge into the UQ process through extracting keywords and evaluating corresponding importance scores. • Empirically, we conduct extensive experiments on the Llama family across five datasets in two tasks and show that CoT-UQ achieves an average AUROC improvement of approximately 5.9% compared to baselines, which verifies the effectiveness of our method. Figure 2: Illustration of our framework, CoT-UQ. Given a question and an incorrect response generated by an LLM, the top of the figure shows two common UQ strategies, which suffer from overconfidence issues. The bottom shows the four-step process of CoT-UQ: performing the reasoning process, extracting step-wise keywords, scoring the importance of keywords relative to the final answer, and leveraging reasoning information to enhance common UQ strategies. CoT-UQ leads to a better-calibrated response-wise uncertainty estimate. 2 Related Works 2.1 Uncertainty Quantification in LLMs Prior efforts to quantify uncertainty and confidence in LLMs can be categorized into four main approaches. The first approach is to derive calibrated confidence by examining agreement across multiple sampled responses ( Malinin and Gales 2021 ; Kuhn et al. 2023 ; Manakul et al. 2023 ; Tian et al. 2023a ) . However, as Qiu and Miikkulainen 2024 recently pointed out, these methods primarily quantify prompt-wise rather than response-wise uncertainty. While Qiu and Miikkulainen 2024 provides a method for response-wise uncertainty, it still relies on generating multiple response samples, making it computationally inefficient. The second approach is to leverage LLM’s own ability to evaluate the confidence of its responses, often through self-probing techniques ( Kadavath et al. 2022 ; Tian et al. 2023b ; Xiong et al. 2023 ) . The third approach is to aggregate token probabilities of its generated response, which includes adopting traditional UQ methods ( Xiao et al. 2022 ; Ye et al. 2024 ) and assigning importance weights to tokens ( Duan et al. 2024a ; Bakman et al. 2024 ) . However, the above two approaches still suffer from overconfidence due to the model’s inherent bias to trust its own outputs ( Mielke et al. 2022 ; Lin et al. 2022

Read the complete text