Process Supervision of Confidence Margin for Calibrated LLM Reasoning

Liaoyaqi Wang, Chunsheng Zuo, William Jurayj, Benjamin Van Durme, Anqi Liu · 2026

Read Process Supervision of Confidence Margin for Calibrated LLM Reasoning by Liaoyaqi Wang, Chunsheng Zuo, William Jurayj, Benjamin Van Durme, Anqi Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23333.txt.

Introduction

Wang and four colleagues introduce Reinforcement Learning with Confidence Margin, a method that jointly optimises correctness and confidence trust. Instead of matching self-belief to success odds, it widens the confidence margin between right and wrong steps within a single reasoning path. A margin-enhanced process reward over mid-budget completions drives the training. Across maths, code, logic, and science benchmarks, calibration gains a lot while accuracy holds or improves. Tuned confidence then enables leaner conformal risk control and confidence-weighted pooling.

Catalog record

CREATORLiaoyaqi Wang, Chunsheng Zuo, William Jurayj, Benjamin Van Durme, Anqi Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33973
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Process Supervision of Confidence Margin for Calibrated LLM Reasoning Liaoyaqi Wang, Chunsheng Zuo, William Jurayj, Benjamin Van Durme, Anqi Liu Abstract Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability. Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation. We introduce Reinforcement Learning with Confidence Margin ( RLCM ), a calibration-aware RL framework that jointly optimizes correctness and confidence reliability via a margin-enhanced process reward over intermediate-budget completions. Rather than aligning confidence to correctness likelihoods, RLCM encourages to widen the confidence margin between correct and incorrect steps within a single reasoning trajectory. Across mathematical, code, logic and science benchmarks, our method substantially improves calibration while maintaining or improving accuracy. We further show that, with calibrated confidence signals, the resulting models enable more efficient conformal risk control and effective confidence-weighted aggregation. 1 Introduction Reasoning large language models (LLMs) can solve substantially harder problems by scaling test-time computation ( Snell et al. 2025 ; Muennighoff et al. 2025 ) , achieving impressive gains across domains such as mathematics, coding and scientific discovery ( Shao et al. 2024 ; OpenAI et al. 2024 ; DeepSeek-AI et al. 2025 ) . However, strong reasoning performance does not guarantee reliable confidence expression ( Huang et al. 2025b ; Yona et al. 2024 ; Jurayj et al. 2025 ) . Figure 1: Accuracy v.s. ECE at the 8k token budget on AIME24. RLCM preserves high accuracy while substantially improving calibration, achieving the strongest Pareto trade-off among the compared methods. This limitation is partly rooted in outcome-driven training objectives, which optimize final answer correctness but do not explicitly encourage the model’s confidence to reflect its true probability of being correct ( Kalai et al. 2025 ; Mei et al. 2026 ) . Moreover, because supervision is concentrated on the final outcome, these objectives provide only weak guidance for how confidence should evolve throughout the reasoning process ( Lightman et al. 2024 ; Luo et al. 2025b ) . As a result, reasoning models may be confidently wrong, and their expressed uncertainty may not faithfully reflect the quality of the reasoning process and final answers ( Lacombe et al. 2025 ) . Misplaced confidence hinders deployment in high-stakes contexts. Uncertainty is a widely used upstream signal for decision making: it can govern whether the model should stop early, revise its answer, invoke external tools, or defer to human experts ( Wu et al. 2025 ; Jiang et al. 2025 ; Ni et al. 2024 ; Strong et al. 2025 ) . Unreliable confidence therefore limits the safe deployment of reasoning models, especially in high-stakes domains and makes it difficult to steer model behavior through uncertainty ( Band et al. 2024 ; Liu et al. 2025b ) . Existing works training language models to express uncertainty, whether based on answer consistency ( Huang et al. 2025a ) , verbalized confidence ( Leng et al. 2025 ) , or post-hoc probing ( Zhang et al. 2025a ) , largely focus on final-answer uncertainty and yield limited control over how confidence evolves across intermediate reasoning steps, leaving process-level calibration largely underexplored. To address this, we propose RLCM (Reinforcement Learning with Confidence Margin), a calibration-aware RL framework that supervises confidence throughout the reasoning trajectory with a margin-based process reward. As is shown in the Figure 2 , we incentivize a model to increase the confidence gap ( Chhikara 2025 ) between correct and incorrect intermediate reasoning steps within the same trajectory, providing more useful scores for downstream calibration. To make such supervision available during training, we attach a lightweight confidence probe to intermediate reasoning states and use it to estimate how likely a truncated reasoning prefix is to generate a correct final answer. This relative margin-enhanced objective provides a more robust calibration signal than direct score matching while remaining compatible with final-answer optimization. By imposing this margin reward at intermediate reasoning steps during the on-policy training stage, we facilitate flexible and cheap post-hoc calibration to address diverse efficiency and safety objectives. Across mathematical reasoning benchmarks of varying difficulty, and on out-of-domain tasks such as coding, scientific, and logic question answering, RLCM improves calibration while preserving or even improving reasoning accuracy, as we indicated in the Figure 1 . These improvements also translate to downstream decision making: calibrated confidence enables more effective conformal risk control, which enforces target error guarantees, and stronger confidence-weighted aggregation results. Figure 2: Overview of RLCM . We select intermediate compute budgets b_{1},\dots,b_{n} along a reasoning trajectory. At each budget, Monte Carlo sampling estimates intermediate correctness, while a lightweight probe predicts confidence from the hidden state. Instead of directly aligning confidence with correctness, RLCM encourages a larger confidence margin between correct and incorrect intermediate states within the same trajectory. 2 Background and Related Work 2.1 Language Model Reinforcement Learning Reinforcement Learning with Verifiable Rewards. Recent progress in reasoning language models has been driven by reinforcement learning with verifiable rewards (RLVR) ( Lambert et al. 2025 ) , exemplified by Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ; DeepSeek-AI et al. 2025 ) . Given a prompt x , the policy \pi_{\theta} samples a group of completions \{y_{1},y_{2},...y_{i}\} , each receiving a scalar reward R_{i} by verifying its predicted answer with the ground-truth answer. GRPO optimizes: \displaystyle\mathcal{J}_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{x,y}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\min\!\Big(r_{i,t}(\theta)\hat{A}_{i},\;\mathrm{clip}\!\big(r_{i,t}(\theta),1-\epsilon,1+\epsilon\big)\hat{A}_{i}\Big)-\beta\,D_{\mathrm{KL}}\!\left(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\right)\Bigg], where r_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid x,y_{i,<t})},\qquad\hat{A}_{i}=\frac{R_{i}-\mathrm{mean}(\{R_{j}\}_{j=1}^{G})}{\mathrm{std}(\{R_{j}\}_{j=1}^{G})}. By using a group-relative advantage as a surrogate for the value function, GRPO training is more efficient, obviating the need for a separate value model like in PPO ( Schulman et al. 2017 ) . However, optimizing exclusively for verifiable correctness often triggers calibration degeneration ( Luo et al. 2025a ) , as binary rewards incentivize models to prioritize correctness over honesty ( Kalai et al. 2025 ; Bereket & Leskovec 2025 ) . This necessitates training strategies that harmonize outcome rewards with internal uncertainty ( Xu et al. 2025 ) , motivating our design of an auxiliary calibration reward. We also remove the KL regularizer following ( Yu et al. 2025 ; Zhang et al. 2026 ) . Process Rewards. RLVR suffers from sparse supervision since rewards are usually assigned only at the end of a rollout. To improve credit assignment, recent work introduces process rewards that provide denser intermediate feedback ( Zheng et al. 2025 ) . Such rewards can be derived from human annotations or LLM judgments for mathematical reasoning ( Yu et al. 2024 ; Wang et al. 2024b ; Zhang et al. 2025b ) , from LLM-based evaluation of tool utility in agentic tasks ( Goldie et al. 2025 ) , or from Monte Carlo completions sampled from intermediate states ( Li et al. 2025a ; Qi et al. 2025 ; Qu et al. 2025 ) . Another line of work uses stronger models to score or distill token-level reasoning traces, including on-policy distillation ( Agarwal et al. 2024 ) , and recent self-distillation variants ( Zhao et al. 2026 ; Hübotter et al. 2026 ) . These approaches, however, often require costly external model calls. In contrast, RLCM relies on a lightweight probe to deliver efficient process-level supervision. 2.2 Uncertainty and Calibration in LLMs Uncertainty Estimation. Transformer-based language

Read the complete text