Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning

Jiuzhou Han, Wray Buntine, Ehsan Shareghi · 2025

Read Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning by Jiuzhou Han, Wray Buntine, Ehsan Shareghi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01773.txt.

Introduction

Jiuzhou Han, Wray Buntine, and Ehsan Shareghi use uncertainty to build training data for process reward models in math reasoning, in a 2025 paper. Language models make errors across the steps of a long solution, and reward models that grade each step can help. Training those graders needs step-level data that is costly to make by hand.

Their framework automates both the generation and labeling of such data, guided by uncertainty. They also propose two ways to merge model outputs that mix majority voting with reward scores. Tests on standard math benchmarks showed the data method works well and the merging methods lifted reasoning scores across several reward models.

Catalog record

CREATORJiuzhou Han, Wray Buntine, Ehsan Shareghi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01773

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning Jiuzhou Han, Wray Buntine, Ehsan Shareghi Abstract Large language models have demonstrated remarkable capabilities in complex mathematical reasoning tasks, but they inevitably generate errors throughout multi-step solutions. Process-level Reward Models (PRMs) have shown great promise by providing supervision and evaluation at each intermediate step, thereby effectively improving the models’ reasoning abilities. However, training effective PRMs requires high-quality process reward data, yet existing methods for constructing such data are often labour-intensive or inefficient. In this paper, we propose an uncertainty-driven framework for automated process reward data construction, encompassing both data generation and annotation processes for PRMs. Additionally, we identify the limitations of both majority vote and PRMs, and introduce two generic uncertainty-aware output aggregation methods: Hybrid Majority Reward Vote and Weighted Reward Frequency Vote, which combine the strengths of majority vote with PRMs. Extensive experiments on ProcessBench, MATH, and GSMPlus show the effectiveness and efficiency of the proposed PRM data construction framework, and demonstrate that the two output aggregation methods further improve the mathematical reasoning abilities across diverse PRMs. The code and data will be publicly available at https://github.com/Jiuzhouh/UnPRM . 1 Introduction Inference-time scaling ( Lightman et al. 2024 ) offers a practical approach to improving reasoning performance of Large Language Models (LLMs) by leveraging increased computational resources during inference. Within this framework, Process Reward Models (PRMs) ( Uesato et al. 2022 ) have been introduced to assess the correctness of intermediate reasoning steps, providing a fine-grained mechanism for scoring and filtering candidate solutions and thereby supporting more robust answer selection. Unlike Outcome Reward Models (ORMs) ( Cobbe et al. 2021 ) , which focus solely on the final answer, PRMs provide fine-grained verification of the entire reasoning process. This granularity allows PRMs to identify solutions in which correct final answers are arrived at via flawed intermediate steps, revealing otherwise undetected errors in reasoning. A major challenge in developing effective PRMs lies in the annotation of high-quality step-level supervision data, which is typically expensive and time-consuming. Early approaches ( Lightman et al. 2024 ) relied on human annotation to ensure label quality, but this method is costly and not scalable. To improve efficiency, recent research has focused on automated annotation techniques, including Monte Carlo (MC) methods that estimate step correctness based on the probability of reaching the correct final answer ( Wang et al. 2024 ; Luo et al. 2024 ) and approaches that leverage strong LLMs as judges of step correctness ( Tan et al. 2025 ; Gao et al. 2024 ) . However, these methods can require substantial computational resources and often suffer from inefficiency. To overcome these challenges, we propose an uncertainty-driven PRM data construction framework. Our approach first generates candidate solutions for annotation using uncertainty-guided sampling, then applies an uncertainty-driven automated annotation process to efficiently and accurately label the correctness of each reasoning step. This pipeline significantly improves both the quality and scalability of PRM training data. While PRMs and Majority Vote ( Wang et al. 2023 ) are commonly used for aggregating LLM outputs, both have notable limitations. Majority Vote, which selects the most frequent answer among sampled outputs, can fail when answers are highly dispersed or when the model confidently produces incorrect solutions, leading to erroneous consensus. Conversely, PRMs may select suboptimal answers when faced with out-of-distribution or particularly challenging problems. To address these limitations and improve output aggregation, we further propose two uncertainty-aware hybrid strategies: Hybrid Majority Reward (HMR) Vote and Weighted Reward-Frequency (WRF) Vote. These methods combine the implicit confidence signals of Majority Vote with the explicit, step-level feedback from PRMs, aiming to achieve more reliable answer selection. We construct several variations of PRM training data and conduct extensive experiments on ProcessBench ( Zheng et al. 2024 ) , MATH ( Hendrycks et al. 2021 ) , and GSMPlus ( Li et al. 2024 ) to validate the efficiency and effectiveness of our uncertainty-driven PRM data construction framework. Additionally, we demonstrate that our proposed uncertainty-aware aggregation strategies generalise well across different PRMs and yield notable performance improvements compared to majority vote and traditional PRM methods, with WRF generally proving to be the more robust strategy. 2 Related Work PRMs in Mathematical Reasoning Mathematical reasoning tasks are among the most challenging for LLMs, as they demand rigorous multi-step logical thinking and precise manipulation of mathematical symbols, leaving little room for ambiguity or guesswork. A single error at any step can invalidate the entire solution, and the abstract, concise nature of mathematical language further compounds the difficulty. To assess the correctness of these intermediate steps, PRMs ( Lightman et al. 2024 ; Uesato et al. 2022 ) have been introduced. PRMs assign a score to each reasoning step, reflecting its likelihood of correctness. By aggregating these step scores, a final score for the entire solution can be obtained, providing an overall assessment of its validity. Furthermore, PRMs enable the selection of the highest-scoring solution from multiple candidates, thereby further improving the reasoning performance of LLMs. Mathematical Reasoning Step Verification Although PRMs demonstrate significant potential for enhancing the performance of LLMs, their effectiveness relies heavily on the availability of high-quality labelled training data, particularly for evaluating the correctness of intermediate reasoning steps. There are three primary approaches to data annotation: (1) Human annotation ( Lightman et al. 2024 ) , which can yield highly accurate labels but is labour-intensive, expensive, and time-consuming; (2) LLM-as-a-Judge methods ( Tan et al. 2025 ; Zhang et al. 2025a ; Gao et al. 2024 ) , which prompt LLMs to directly assess step correctness, offering automation but incurring high computational costs, especially when labelling large datasets, and inheriting the limitations of LLMs such as hallucination and inaccuracies; and (3) inferring step correctness from solution outcomes using techniques such as Monte Carlo estimation. The latter enables automated annotation by leveraging open-source LLMs ( Wang et al. 2024 ) , but often suffers from inefficiency. To address this, Luo et al. 2024 introduced a more efficient Monte Carlo Tree Search algorithm using binary search, while Sun et al. 2025 further improved efficiency with an adaptive binary search method. Building on these developments, we propose an uncertainty-driven search algorithm for automated step label annotation, further enhancing annotation efficiency without sacrificing quality. Uncertainty Estimation in LLMs Uncertainty estimation plays a crucial role in assessing the confidence of outputs generated by LLMs. Logits-based methods ( Han, Buntine, and Shareghi 2024 ; Yang et al. 2023 ) quantify uncertainty by analysing the token-level logits produced by the model; however, these approaches are only feasible when direct access to output token logits is available. Alternatively, self-consistency ( Wang et al. 2023 ) provides an implicit measure of model confidence by sampling multiple outputs for a given question—answers that appear frequently among the samples are considered high-confidence, while those with low frequency indicate greater uncertainty. Prior studies ( Han, Buntine, and Shareghi 2024 ; Zhao et al. 2025 ) have found a strong correlation between higher model confidence and answer correctness, with errors being more likely when the model is uncertain. Motivated by these findings, we propose leveraging uncertainty as a guiding signal to identify erroneous steps in mathematical reasoning solutions, thereby improving the

Read the complete text