Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads
Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan · 2025
Read Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads by Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06209.txt.
Introduction
Jingwei Ni and colleagues present ReProbe, a light way to check each step of a language model's reasoning as it is produced. Checking steps can raise accuracy, but standard reward models are costly and need large amounts of labeled data. ReProbe instead reads the model's own internal states while it generates text.
A small transformer probe, trained on those internal states, estimates how credible each reasoning step is. Labels can come from a larger model or from the model itself in a self-supervised way. With under ten million parameters, the probes matched or beat reward models up to eight hundred ten times larger across math, planning, and question answering.
Catalog record
| CREATOR | Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan Abstract Solving complex tasks usually requires LLMs to generate long multi-step reasoning chains. Previous work has shown that verifying the correctness of individual reasoning steps can further improve the performance and efficiency of LLMs on such tasks and enhance solution interpretability. However, existing verification approaches, such as Process Reward Models (PRMs), are either computationally expensive, limited to specific domains, or require large-scale human or model-generated annotations. Thus, we propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty estimation. We train transformer-based uncertainty quantification heads (UHeads) that use the internal states of the frozen LLM to estimate the uncertainty of its reasoning steps during generation. The approach is fully automatic: target labels are generated either by another larger LLM (e.g., DeepSeek R1) or in a self-supervised manner by the original model itself. UHeads are both effective and lightweight, containing less than 10M parameters. Across multiple domains, including mathematics, planning, and general knowledge question answering, they match or even exceed the performance of PRMs that are up to 810× larger. Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification, offering a promising direction towards scalable and generalizable introspective LLMs. † † ♢ \diamondsuit Equal contribution. Correspond to {jingni,efadeeva,msachan}@ethz.ch, [email protected], [email protected] 1 Introduction Chain-of-thought (CoT) prompting has proven highly effective in eliciting the reasoning capabilities of large language models (LLMs) for solving complex tasks ( 48 ) . Recent post-training approaches further enhance this ability through reinforcement learning, rewarding models for generating responses that conform to the CoT pattern and yielding correct final answers ( 11 ; 52 ; 1 ) . However, the emphasis on verifying only final answers raises concerns about the reliability of intermediate reasoning steps ( 33 ; 61 ; 8 ) . Flawed steps can propagate and systematically distort conclusions ( 17 ) , while in some cases LLMs may still produce the correct answer despite erroneous intermediate steps ( 4 ) . Such issues undermine trust in CoT-based methods, particularly in high-stakes domains like medicine ( 3 ) and law ( 15 ) . Figure 1: Process reward models (top) in comparison to uncertainty quantification heads (bottom). The contemporary approach to supervise models for step-level correctness is through process reward models (PRMs: 33 ; 59 ). However, PRM-based supervision faces key drawbacks. First, it requires expensive monte-carlo rollouts for training data annotation and deploying an additional LLM supervisor, adding substantial computational overhead. Second, this approach fails in tasks like mathematical proofs ( 7 ) and planning ( 62 ) , where step-level correctness cannot be inferred from the final result alone. Another promising line of work for assessing the correctness of reasoning steps leverages uncertainty quantification (UQ) methods ( 18 ; 39 ) . Unlike PRMs, which rely on external verification, UQ assumes that a model’s outputs and internal states provide information about the reliability of its generations. To date, only relatively simple unsupervised UQ methods have been investigated for LLM reasoning ( 17 ; 51 ) . These approaches are primarily based on consistency-based heuristics such as majority voting and rely on simplistic uncertainty estimates without leveraging any model internals. While computationally lightweight, they consistently fail to capture subtle reasoning flaws and lag significantly behind PRMs in performance. In this work, we ask: Can the cost-efficiency of UQ methods be combined with the performance of PRMs? Recent research in UQ shows that data-driven approaches substantially outperform unsupervised ones in detecting factual hallucinations ( 6 ; 9 ; 42 ) . Building on this insight, we introduce a lightweight data-driven uncertainty-based verifier for step-level reasoning. Specifically, we adapt the UHead framework ( 42 ) to train uncertainty quantification heads on top of frozen LLMs for reasoning step verification. Unlike PRMs, which rely on expensive rollouts and training of a large supervisor LLM, UHeads exploit internal LLM states (attention and token probabilities) as features, making them far more efficient (see fig. 1 ): UHeads trained in our work require less than 10M parameters. Training labels are generated automatically, either by using a powerful external LLM (e.g., DeepSeek R1) as a judge or by relying on the original LLM itself. This enables scalable supervised or self-supervised training without the need for verifiable final answers, human annotations, or costly Monte Carlo rollouts. Extensive in-domain (ID) and out-of-domain (OOD) evaluation highlights UHeads’ advantages in three key aspects: (1) Generalizable Performance : UHeads achieve competitive or superior performance compared to PRM baselines in step-level verification and test-time scaling, including best-of- N selection and step-level correctness supervision, especially when dealing with OOD reasoning tasks (e.g., reasoning-intensive QA and planning); (2) Computational Efficiency : UHeads substantially outperform PRMs that are 150 \times larger, while remaining competitive with PRMs up to 800 \times larger; and (3) Training Data Efficiency : unlike PRMs, which often rely on proprietary datasets ( 24 ) , costly human annotations ( 33 ) , or consensus-filtering pipelines involving both Monte Carlo rollouts and LLM judgments ( 59 ) , UHeads can be trained on data annotated automatically in a self-supervised manner. Our contributions are the following: \bullet We introduce a lightweight approach to verifying step-level correctness in LLM-generated reasoning by extending the UHead framework. Our 9.8M-parameter UHead achieves competitive or superior performance compared to 7B-scale PRMs across diverse reasoning tasks and settings, while being significantly more efficient at inference time in terms of memory and compute. \bullet We develop an efficient and scalable data annotation pipeline that eliminates the need for human-labeled data, verifiable final answers, or costly Monte Carlo rollouts. We further show that high performance in reasoning verification can be achieved using annotations derived either from external verification models (e.g., DeepSeek R1) or from a fully self-supervised process. \bullet We show that our approach achieves strong generalization across domains: a single UHead trained only on mathematical tasks generalizes effectively to planning and general knowledge QA. Unlike PRMs, which often overfit to domain-specific reasoning patterns, our lightweight UHead verifier leverages internal LLM uncertainty signals, enabling robust reasoning verification without requiring domain adaptation. 2 Background 2.1 Test-Time Scaling State-of-the-art LLMs solve complex tasks by generating CoT reasoning traces ( 22 ) . Given an input question \mathbf{x} , models produce a sequence of reasoning steps \mathbf{r}=\{r_{1},r_{2},\dots,r_{T}\} followed by a final answer \mathbf{y}=g(\mathbf{r},\mathbf{x}) . Formally, an LLM parameterized by \theta models the conditional distribution: P_{\theta}(\mathbf{y},\mathbf{r}\mid\mathbf{x})=\prod_{t=1}^{T}P_{\theta}(r_{t}\mid\mathbf{x},\mathbf{r}_{<t})\cdot P_{\theta}(\mathbf{y}\mid\mathbf{x},\mathbf{r}), where r_{t} denotes the t -th reasoning step and \mathbf{r}_{<t} are the preceding steps. Although CoT reasoning improves final task performance, locally incorrect steps r_{t} can propagate and cause wrong answers. Test-time scaling (TTS) algorithms ( 54 ; 43 ) aim to further improve the performance of LLMs without retraining the entire model, by allocating additional computation or using more sophisticated inference strategies. By leveraging multiple candidate solutions and verification signals, TTS facilitates selecting a correct answer. Common approaches include online and offline best-of- N (BoN) sampling. Other techniques employ sophisticated search and sampling techniques based on beam search and backtracking, such as the tree of