Improving Value-based Process Verifier via Low-Cost Variance Reduction
Zetian Sun, Dongfang Li, Baotian Hu, Min Zhang · 2025
Read Improving Value-based Process Verifier via Low-Cost Variance Reduction by Zetian Sun, Dongfang Li, Baotian Hu, Min Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10539.txt.
Introduction
Zetian Sun and co-authors sharpen in 2025 the judges that grade AI reasoning step by step. A value-based judge estimates the chance that a half-finished solution still reaches the right answer. Training such judges is costly, because each estimate needs many sampled completions, and the noise in those estimates weakens the judge.
The team shows the noise is mostly variance, not bias. Their fix blends the estimates from the current step and the next one, which provably trims the variance at no extra cost. On two math test sets the improved judge beat the plain one and a regression-based rival. Better step scores mean better search over reasoning paths.
Catalog record
| CREATOR | Zetian Sun, Dongfang Li, Baotian Hu, Min Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10539
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Value-based Process Verifier via Low-Cost Variance Reduction Zetian Sun, Dongfang Li, Baotian Hu, Min Zhang Abstract Large language models (LLMs) have achieved remarkable success in a wide range of tasks. However, their reasoning capabilities, particularly in complex domains like mathematics, remain a significant challenge. Value-based process verifiers, which estimate the probability of a partial reasoning chain leading to a correct solution, are a promising approach for improving reasoning. Nevertheless, their effectiveness is often hindered by estimation error in their training annotations, a consequence of the limited number of Monte Carlo (MC) samples feasible due to the high cost of LLM inference. In this paper, we identify that the estimation error primarily arises from high variance rather than bias, and the MC estimator is a Minimum Variance Unbiased Estimator (MVUE). To address the problem, we propose the Com pound M onte C arlo S ampling (ComMCS) method, which constructs an unbiased estimator by linearly combining the MC estimators from the current and subsequent steps. Theoretically, we show that our method leads to a predictable reduction in variance, while maintaining an unbiased estimation without additional LLM inference cost. We also perform empirical experiments on the MATH-500 and GSM8K benchmarks to demonstrate the effectiveness of our method. Notably, ComMCS outperforms regression-based optimization method by 2.8 points, the non-variance-reduced baseline by 2.2 points on MATH-500 on Best-of-32 sampling experiment. 1 Introduction Figure 1: Illustration of the estimation variance across different ground-truth values. We compare variances with number of trials ranging from \{8,10,16\} , and variance applying ComMCS. The variance after using our method (8-trials, ComMCS) is approximately the variance obtained by using 25% more sampling samples (10-trials). In recent years, large language models (LLMs) have demonstrated remarkable capabilities across various reasoning tasks that require complex multi-step reasoning, such as mathematics and programming ( Hurst et al. 2024 ; Yang et al. 2024a ; Yang et al. 2024b ; Dubey et al. 2024 ) , yet they still make mistakes when solving challenging problems. To address this issue, verification-based method has recently emerged to improve LLM reasoning ( Uesato et al. 2022 ; Lightman et al. 2024 ) . In general, verification models are trained to evaluate and potentially correct the reasoning trajectories during the reasoning process, which can ensure higher accuracy and consistency in LLM outputs by re-ranking candidate responses. They can also provide valuable feedback for further improvement of LLMs ( Li et al. 2023 ; Wu et al. 2023 ) . Verification-based methods can be categorized by annotation granularity and strategy, with the value-based process verifier being one approach in this classification. Specifically, value-based process verifier is a kind of process-supervised verifier , as the training annotations are based on evaluations of each reasoning steps. Value-based process verifier is also a kind of value-based verifier , as the verifier aims at predicting the success rate of each reasoning steps, which is the empirical probability of this step leading to the correct final answer in mathematical reasoning scenario, i.e., the value of current step. The training data of value-based process verifier mainly relies on MC estimation. Typically, multiple reasoning trajectories are independently generated by LLMs, and the success rate is computed as the average outcome correctnesses across all trajectories. Outcome correctness is verified through reliable rule-based matching. However, the value estimation is inaccurate due to the limitation of sampling size, which is restricted due to the high cost of LLM inference ( Zhang et al. 2025b ) . This motivates us to study the following research question: is it possible to reduce the estimation error when estimating values without introducing additional LLM inference overhead? In this paper, we present an analysis about the MC estimation error in mathematical reasoning scenario from the perspective of Markov Decision Process (MDP) in § 4.1 . We build a bridge between the MC estimation and the Binomial distribution, and conclude that the MC estimation is the Minimal Variance Unbiased Estimator (MVUE). The conclusion implies that estimation error is due to estimation variance instead of biased estimation, and the variance cannot be reduced without introducing additional information. Based on the analysis, we propose ComMCS in § 4.2 , an unbiased estimation method that utilizing compound MC sampling results to reduce variance without additional LLM inference overhead. This core idea is conceptually analogous to variance reduction techniques in reinforcement learning, such as Temporal Difference (TD) learning, where future value estimates are used to update current ones. We adapt this principle to the context of LLM verifiers. Specifically, we quantify the form of the variance of MC estimation that is based on current step, and the compound variance of MC estimation that is based on the subsequent few steps. Through theoretical analysis, we show that (1) the linear combination of MC estimations of subsequent few steps is also an unbiased estimation of the value of current step , and (2) under specific conditions, the estimation variance can be reduced . Typically, the condition is related to the distribution of future values and the coefficients of linear combinations. We provide the practical implementation of ComMCS in § 4.3 , where we narrow down the linear combination into two items, the current step and its next step. We make several approximations. First of all, we use a categorical distribution to estimate the one-step value distribution, which models the shape of distribution of next step value and is used to estimate the variances. The categorical distribution is modeled by the output of our value-based process verifier, and the verifier is then optimized by cross entropy loss. Secondly, we assume that the one-step value distribution follows the Gaussian distribution class. Finally, we use the MC estimations of values as the proxy of true values to estimate the variances. These practical approximations allow us to calculate and compare variances, after which we find the suitable coefficients for items in the linear combination heuristically and optimize verifier iteratively. The simulated effect of our variance reduction method is shown in Figure 1 , where we assume value distribution follows Gaussian distribution. We then perform extensive experiments on two mathematical reasoning tasks: MATH-500 ( Lightman et al. 2024 ) and GSM8K ( Cobbe et al. 2021 ) to demonstrate the effectiveness of our method across different model series and search strategies. Compared with verifiers that preform return distribution modeling or regression, the value distribution modeling method can achieve comparable performances. After performing our method, we observe a consistent improvement across different settings. For example, our method trained on Deepseek-math-7b-instruct outperforms baselines by 2-3 points on Math-500 on Best-of-N sampling. In beam search experiments, our method trained on Deepseek-math-7b-instruct outperforms other baselines by 1-2 points on MATH-500. Our contributions are summarized as follows: • We are the first to systematically identify and address the high variance of the MC estimator as the key bottleneck limiting the performance of value-based process verifiers. • We propose ComMCS , a theoretically-grounded method that reduces variance by compounding estimators from subsequent steps without extra LLM inference cost. • Our extensive experiments show that ComMCS achieves improved performance over baseline verifiers and demonstrate the potential for diversification in modeling the objective of value-based process verifiers. 2 Related Work 2.1 Test-time Scaling for Math