SelfDoubt : Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio

Satwik Pandey, Suresh Raghu, Shashwat Pandey · 2026

Read SelfDoubt : Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio by Satwik Pandey, Suresh Raghu, Shashwat Pandey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06389.txt.

Introduction

Satwik Pandey, Suresh Raghu, and Shashwat Pandey propose SelfDoubt, a single-pass uncertainty framework. Sampling costs too much, single-pass proxies vary, and proprietary APIs hide probabilities, leaving no reliable signal. SelfDoubt reads signals from the reasoning trace. Its Hedge-to-Verify Ratio spots uncertainty markers and weighs self-checking against them. Across seven models and three benchmarks, traces with no hedging markers prove correct 96 percent of the time, a free gate. The full score beats sampling entropy at 10 times lower cost. A cascade reaches 90 percent accuracy at 71 percent coverage.

Catalog record

CREATORSatwik Pandey, Suresh Raghu, Shashwat Pandey
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57149
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06389

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SelfDoubt : Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio Satwik Pandey, Suresh Raghu, Shashwat Pandey Abstract Uncertainty estimation for reasoning language models remains difficult to deploy in practice: sampling-based methods are computationally expensive, while common single-pass proxies such as verbalized confidence or trace length are often inconsistent across models. This problem is compounded for proprietary reasoning APIs that expose neither logits nor intermediate token probabilities, leaving practitioners with no reliable uncertainty signal at inference time. We propose SelfDoubt , a single-pass uncertainty framework that resolves this impasse by extracting behavioral signals directly from the reasoning trace itself. Our key signal, the Hedge-to-Verify Ratio ( HVR \mathrm{HVR} ), detects whether a reasoning trace contains uncertainty markers and, if so, whether they are offset by explicit self-checking behavior. Unlike methods that require multiple sampled traces or model internals, SelfDoubt operates on a single observed reasoning trajectory, making it suitable for latency- and cost-constrained deployment over any proprietary API. We evaluate SelfDoubt across seven models and three multi-step reasoning benchmarks (BBH, GPQA-Diamond, and MMLU-Pro). Most notably, traces containing no hedging markers are correct 96% of the time, revealing an emergent high-precision confidence gate at zero additional cost. For the remaining cases, the full SelfDoubt score significantly outperforms sampling-based semantic entropy at 10 × \times lower inference cost. A deployment cascade combining both stages attains 90% accuracy at 71% coverage without any task-specific labels. These results establish SelfDoubt as a scalable, production-ready foundation for uncertainty estimation over proprietary reasoning models. All code is publicly available at https://github.com/satwik2711/SelfDoubt . 1 Introduction Uncertainty quantification for reasoning language models remains caught between two unappealing extremes. Sampling-based methods, including Semantic Entropy ( 9 ; 5 , SE;) , P(True) self-evaluation ( 7 ) , and semantic-geometric approaches ( 10 ; 16 ) , require N independent forward passes, which is prohibitive when a single reasoning pass already costs thousands of tokens. Single-pass alternatives exist, but each has a limiting tradeoff: verbalized confidence ( 21 ; 27 ) is miscalibrated on hard tasks and weaker models; trace length ( 4 ) correlates with uncertainty only on intermediate-difficulty benchmarks; and probe-based methods ( 8 ) require hidden-state access. A further constraint: commercially deployed reasoning APIs do not expose log-probabilities on reasoning tokens, making methods that depend on token-level probabilities ( 11 ; 28 ) architecturally ineligible where UQ is most needed. We propose SelfDoubt , a single-pass uncertainty framework that extracts behavioral signals directly from the reasoning trace. Its key signal, the Hedge-to-Verify Ratio (HVR), is a single scalar: the number of hedge markers divided by the number of verify markers plus one. Intuitively, hedging expresses doubt ( “maybe,” “perhaps,” “not sure” ), whereas verification acts on it ( “let me check,” “verify,” “substitute back” ). HVR therefore captures whether expressed doubt is resolved or left open. Z-score fusion of HVR with verbalized confidence yields the full SelfDoubt score, requiring zero training, no model internals, and only 90 unlabeled calibration traces per model. 1. HVR = 0 gate. We introduce a zero-cost correctness filter requiring only regex matching against a marker dictionary. Traces with zero hedging language are correct 96.1% of the time (25.4% coverage, 0.58% genuine error rate after label-noise correction) across 7 models and 3 datasets. 2. SelfDoubt . We present an O(1) uncertainty score fusing HVR with verbalized confidence via z-score normalization. SelfDoubt significantly outperforms Semantic Entropy on discrimination ( p{=}0.001 ) while matching its selective prediction quality at 10\times lower inference cost. 3. Unsupervised marker discovery. We introduce an unsupervised per-model pipeline that builds model-specific dictionaries from 90 unlabeled traces: no correctness labels, no manual curation, and no retraining when switching models. 4. Production deferral cascade. Tier 1 (HVR = 0 gate) plus Tier 2 (calibrated z-sum threshold) yields 71% coverage at 89.7% accuracy, a +9.2 pt lift requiring 4 stored scalars per model. (a) Mean AUROC and AURAC across all 21 runs Figure 1: Headline results. (a) Across 21 runs, SelfDoubt is the only method that leads both metrics at O(1) cost: mean AUROC 0.7895 and mean AURAC 0.8992 (slightly above SE at 0.8988). 2 Related Work Method landscape: Table 1 organizes UQ methods along the axes critical for reasoning-model deployment: computational cost and method requirements. Table 1: Uncertainty quantification methods for language models. O(1) methods above the first rule; O(N) and O(N^{2}) below. Method Cost Requirements Verb. Conf. ( 21 ; 27 ) O(1) Prompt mod. Trace Length ( 4 ) O(1) Trace TL+VB ( 4 ) O(1) Trace + prompt mod. Lex. Hints ( 22 ) O(1) Trace + sentiment scorer CoT-UQ ( 28 ) O(1) Token logits EPR/WEPR ( 11 ) O(1) Top- K logprobs SEP ( 8 ) O(1) Hidden states Latent CoE ( 23 ) O(1) Hidden states P(True) self-eval. ( 7 ) O(N) Self-eval + sampled refs (often logprobs) SE ( 9 ; 5 ) O(N) N samples + NLI SV ( 10 ) O(N) N samples + log-det(Gram) GEO ( 16 ) O(N) N samples + convex hull Topo-UQ ( 3 ) O(N^{2}) N samples + GED SelfDoubt \mathbf{O(1)} Trace + Prompt mod. Sampling-based UQ: Semantic Entropy ( 9 ; 5 ) clusters N sampled outputs by meaning; Semantic Volume ( 10 ) and Geometric Uncertainty ( 16 ) measure embedding spread; Topo-UQ ( 3 ) computes graph edit distance at O(N^{2}) cost. All require 10–20 \times inference cost, which is particularly prohibitive for reasoning models where a single pass already costs thousands of tokens. Trace-based O(1) methods: 4 showed trace length is an emergent uncertainty signal, and TL+VB improves over either component alone. 27 found reasoning models are better calibrated when verbalizing confidence. The closest concurrent work is 22 , who showed hedging words in CoT traces reduce accuracy by up to 40% relative. 17 further showed that reasoning traces surface self-confidence signals absent in direct-answer generations. Probe-based and hidden-state methods: Probe-based methods (SEP ( 8 ) , Latent CoE ( 23 ) ) require hidden states; logprob methods (EPR/WEPR ( 11 ) , CoT-UQ ( 28 ) ) require token probabilities unavailable from production APIs. Both are included in Table 1 but excluded from comparison; What remains missing is an O(1) method that captures deliberation structure from text alone, without trained components or model internals. 3 SelfDoubt Reasoning traces do more than expose a model’s final answer: they also expose how the model expresses and resolves doubt during reasoning. SelfDoubt turns that behavioral signal into a single-pass uncertainty score. (Figure 2 ). Figure 2: SelfDoubt pipeline: calibration-time marker discovery and inference-time scoring. 3.1 Data-Driven Marker Discovery The central challenge in HVR is not the ratio itself, but defining what should count as hedging and verification across different models. Reasoning models do not express uncertainty through a uniform vocabulary, so a fixed lexicon can miss model-specific phrasing and weaken the signal. We therefore use a two-stage data-driven pipeline: Stage 1 builds seed vocabularies for hedging and verification, and Stage 2 expands those seeds into model-specific marker dictionaries from unlabeled traces. Stage 1: Data-Driven Seed Generation. Rather than hand-specifying initial hedge and verify words, we generate them from model consensus. We query multiple language models to produce candidate single-word lists for two behaviors: language associated with

Read the complete text