Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning

Fei Wu, Zhenrong Zhang, Qikai Chang, Jianshu Zhang, Quan Liu, Jun Du · 2026

Read Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning by Fei Wu, Zhenrong Zhang, Qikai Chang, Jianshu Zhang, Quan Liu, Jun Du (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03823.txt.

Introduction

Fei Wu and five co-authors improve how reinforcement learning credits individual reasoning steps in language models trained on verifiable math rewards. Outcome-only rewards judge the final answer, so models cannot tell necessary deduction from redundant checking. They sometimes keep verifying after solving, and can even talk themselves out of a correct answer into a wrong one.

The team built a training-free probe that reads a model's intermediate confidence and correctness at each step, combining them into a Step Potential signal estimating reasoning progress. Their Step Potential Advantage Estimation then assigns credit using that signal, amplifying the steps that truly advance a solution. Finer credit could make reasoning training cheaper and answers shorter, since models would learn to stop when the work is actually done.

Catalog record

CREATORFei Wu, Zhenrong Zhang, Qikai Chang, Jianshu Zhang, Quan Liu, Jun Du
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03823

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning Fei Wu, Zhenrong Zhang, Qikai Chang, Jianshu Zhang, Quan Liu, Jun Du Abstract Reinforcement Learning with Verifiable Rewards (RLVR) elicits long chain-of-thought reasoning in large language models (LLMs), but outcome-based rewards lead to coarse-grained advantage estimation. While existing approaches improve RLVR via token-level entropy or sequence-level length control, they lack a semantically grounded, step-level measure of reasoning progress. As a result, LLMs fail to distinguish necessary deduction from redundant verification: they may continue checking after reaching a correct solution and, in extreme cases, overturn a correct trajectory into an incorrect final answer. To remedy the lack of process supervision, we introduce a training-free probing mechanism that extracts intermediate confidence and correctness and combines them into a Step Potential signal that explicitly estimates the reasoning state at each step. Building on this signal, we propose Step Potential Advantage Estimation (SPAE), a fine-grained credit assignment method that amplifies potential gains, penalizes potential drops, and applies penalty after potential saturates to encourage timely termination. Experiments across multiple benchmarks show SPAE consistently improves accuracy while substantially reducing response length, outperforming strong RL baselines and recent efficient reasoning and token-level advantage estimation methods. The code is available at https://github.com/cii030/SPAE-RL . Figure 1: Given the same user query, the baseline RLVR model DAPO produces a Right-to-Wrong failure due to over-checking, whereas SPAE terminates confidently after reaching the correct solution. The value with colored background appended after each step indicates the corresponding Step Potential. 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) has become a central paradigm for eliciting long chain-of-thought (CoT) reasoning in large language models (LLMs) ( Wei et al. 2022 ; OpenAI 2024 ; Guo et al. 2025 ; Li et al. 2025 ) . By optimizing outcome-level correctness, RLVR aligns model behavior with verifiable task success and delivers substantial gains on mathematics, logic, and coding benchmarks. However, RLVR provides sparse supervision since reward arrives only after the full generation is complete ( Sun et al. 2025 ) . This outcome-only feedback makes credit assignment ambiguous: the policy cannot reliably identify which parts of a trajectory are essential to reaching the solution and which are merely incidental. In practice, this ambiguity often manifests as unnecessarily long and circuitous reasoning. Recent work mitigates this issue with finer-grained heuristics. One dominant line uses token entropy as a proxy for importance, positing that high-entropy tokens correspond to exploration ( Cheng et al. 2025a ; Chen et al. 2025b ; Wang et al. 2025c ) . Another line regularizes verbosity by rewarding correctness while penalizing length, encouraging early stopping ( Zhang et al. 2025 ; Shen et al. 2025 ; Cheng et al. 2025b ) . These approaches largely operate at the token or sequence level and remain agnostic to semantic progress. Crucially, both categories lack a step-level estimate of reasoning progress that can distinguish necessary deduction from redundant verification. To bridge this gap, we introduce a training-free probing mechanism that makes step-wise progress observable. After each reasoning step, we prompt the model to produce a tentative answer and extract two intermediate signals: confidence and correctness. We then combine them into Step Potential that estimates current reasoning state: high potential indicates justified confidence, while low potential reflects unreliable reasoning. Step Potential enables a direct diagnosis of a failure mode we term Over-Checking . Once the reasoning model has solved the problem (Step Potential saturates), it often continues to generate redundant post-solution verification. More importantly, prolonged verification increases the risk of a Right-to-Wrong (R2W) Failure : after reaching a correct solution, the model continues checking and eventually revises its answer to a wrong one (Figure 1 ). Figure 2 quantifies this phenomenon by separating tokens into solving and checking phases, and reports the R2W rate on incorrect trajectories. Notably, even the latest strong model Qwen3-32B ( Yang et al. 2025 ) produces substantial redundant checking tokens and still exhibits R2W failures. Figure 2: Quantifying Over-Checking on AIME2024 (averaged over 16 samples). Top: average solving and checking tokens on correct responses. Bottom: Right-to-Wrong (R2W) rate on incorrect responses. Building on Step Potential, we propose Step Potential Advantage Estimation (SPAE), an RL method that directly incorporates step-level potential into policy optimization. SPAE improves redundancy control and credit assignment by (1) applying a potential saturation penalty to encourage timely termination once the model has reached a correct solution, and (2) amplifying advantages for pivotal transitions that induce large potential increases, penalizing steps that decrease potential We evaluate SPAE on challenging mathematical benchmarks and out-of-distribution tasks with multiple reasoning models (Qwen and Llama families). SPAE consistently outperforms strong RL baselines and closely related efficient reasoning and token-level advantage estimation methods. Beyond improving accuracy, SPAE effectively prunes redundant post-solution verification, reducing inference cost without sacrificing performance. For example, on AIME2024, AIME2025, and GPQA, SPAE reduces the average response length of DeepSeek-R1-Distill-Qwen-7B by 25.1%, 25.3%, and 24.4%, while improving accuracy by 6.7%, 3.3%, and 1.1%, respectively. Our contributions are summarized as follows: • We introduce a training-free probing mechanism and the Step Potential metric, and formalize Over-Checking as a pathological behavior that degrades efficiency and can trigger Right-to-Wrong failures. • We propose SPAE, a step-aware RL credit assignment method that encourages large gains in Step Potential, penalizes declines, and suppresses redundant post-solution checking. • Extensive experiments show that SPAE simultaneously improves accuracy and reduces inference length across models and benchmarks. Figure 3: Overview of our proposed method. (a) SPAE integrates Step Potential from the probing mechanism into the RLVR optimization loop by combining group-relative outcome advantages with step-aware credit assignment after each rollout. (b) Our training-free probing mechanism estimates Step Potential by inserting a prompt after each reasoning step to compute confidence and correctness from the model’s induced answers; the bottom panel visualizes the resulting Step Potential values as a 2D contour over the (Acc, Conf) space. 2 Preliminary 2.1 Problem Formalization We optimize an LLM policy \pi_{\theta} for mathematical reasoning. Given a query q\sim\mathcal{D} with ground-truth answer y^{*} , the model generates an output o\sim\pi_{\theta}(\cdot\mid q) . In long-CoT models such as DeepSeek-R1 ( Guo et al. 2025 ) and Qwen3 ( Yang et al. 2025 ) , o comprises a reasoning trajectory \tau (typically wrapped by <think> and </think> ) and a final summary s , i.e., o=[\tau;s] . We segment \tau=[\tau^{1},\dots,\tau^{K}] into K discrete reasoning steps, where each step is a contiguous token span separated by explicit delimiters (e.g., “.\n\n”). These steps are the basic units of analysis in this work. RLVR maximizes the expected task-level reward: \mathcal{J}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,o\sim\pi_{\theta}(\cdot|q)}\!\left[R(o,y^{*})\right], (1) where R(o,y^{*})\in\{0,1\} is a binary reward that checks whether the answer extracted from s matches y^{*} . 2.2 Reinforcement Learning with Verifiable Reward Group Relative Policy Optimization (GRPO). To avoid value-network overhead in Proximal Policy Optimization ( Schulman et al. 2017 ) , GRPO ( Shao et al. 2024 ) estimates baselines from group statistics. For each q\sim\mathcal{D} , GRPO samples G responses \{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q) , computes rewards \{R(o_{i},y^{*})\}_{i=1}^{G} , and maximizes: \displaystyle\mathcal{J}_{\text{GRPO}}(\theta) \displaystyle=\mathbb{E}_{q\sim\mathcal{D},\,\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)}\bigg[ (2) \displaystyle\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{j=1}^{|o_{i}|}\min\Big(r_{i,j}(\theta)\,\hat{A}_{i,j}, \displaystyle\operatorname{clip}\!\left(r_{i,j}(\theta),1-\epsilon,1+\epsilon\right)\hat{A}_{i,j}\Big)\bigg], where \epsilon is the clipping threshold, and \displaystyle r_{i,j}(\theta) \displaystyle=\frac{\pi_{\theta}(o_{i,j}\mid q,o_{i,<j})}{\pi_{\theta_{\text{old}}}(o_{i,j}\mid q,o_{i,<j})}, (3) \displaystyle\hat{A}_{i,j}=\hat{A}_{i} \displaystyle=\frac{R(o_{i},y^{*})-\operatorname{mean}(\{R(o_{k},y^{*})\}_{k=1}^{G})}{\operatorname{std}(\{R(o_{k},y^{*})\}_{k=1}^{G})}. GRPO assigns a single scalar advantage \hat{A}_{i} to all tokens in o_{i} , eliminating the critic but yielding coarse

Read the complete text