Towards Robust Process Reward Modeling via Noise-aware Learning
Bin Xie, Bingbing Xu, Xueyun Tian, Yilin Chen, Huawei Shen · 2026
Read Towards Robust Process Reward Modeling via Noise-aware Learning by Bin Xie, Bingbing Xu, Xueyun Tian, Yilin Chen, Huawei Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12748.txt.
Introduction
Bin Xie and four co authors work on process reward models for reasoning tasks. These models score each step of a reasoning chain, not just the final answer. They have achieved strong results on complex reasoning problems. Their bottleneck is the cost of process level supervision, since human step labels are dear.
A widely used alternative estimates step rewards by Monte Carlo sampling. It defines a step reward as the chance that a policy reaches the right answer from that step. Such estimates are noisy, though, and noise can mislead training. The team pursue noise aware learning for these models. Cleaner step signals could make reasoning training both cheaper and better.
Catalog record
| CREATOR | Bin Xie, Bingbing Xu, Xueyun Tian, Yilin Chen, Huawei Shen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12748
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Robust Process Reward Modeling via Noise-aware Learning Bin Xie, Bingbing Xu, Xueyun Tian, Yilin Chen, Huawei Shen Abstract Process Reward Models (PRMs) have achieved strong results in complex reasoning, but are bottlenecked by costly process-level supervision. A widely used alternative, Monte Carlo Estimation (MCE), defines process rewards as the probability that a policy model reaches the correct final answer from a given reasoning step. However, step correctness is an intrinsic property of the reasoning trajectory, and should be invariant to policy choice. Our empirical findings show that MCE producing policy-dependent rewards that induce label noise, including false positives that reward incorrect steps and false negatives that penalize correct ones. To address above challenges, we propose a two-stage framework to mitigate noisy supervision. In the labeling stage, we introduce a reflection-aware label correction mechanism that uses a large language model (LLM) as a judge to detect reflection and self-correction behaviors related to the current reasoning step, thereby suppressing overestimated rewards. In the training stage, we further propose a N oise- A ware I terative T raining framework that enables the PRM to progressively refine noisy labels based on its own confidence. Extensive Experiments show that our method substantially improves step-level correctness discrimination, achieving up to a 27% absolute gain in average F1 over PRMs trained with noisy supervision. † † footnotetext: ∗ Corresponding author. 1 Introduction Outcome reward models (ORMs) have driven remarkable success in advancing large language model (LLM) reasoning 2 ; 21 ; 25 ; 10 . However, they suffer from reward sparsity and struggle with credit assignment, leading to inefficient exploration and reward hacking in both reinforcement learning (RL) and test-time scaling 23 ; 16 ; 18 ; 4 ; 5 ; 20 . Process reward models (PRMs) address these issues by providing step-by-step feedback, a technique that has been proven to be highly effective for complex reasoning tasks 6 ; 26 ; 29 ; 12 ; 27 . PRMs, when trained on extensive labeled datasets, function as discriminative classifiers that evaluate the correctness of intermediate reasoning steps 26 ; 19 ; 34 . However, a fundamental bottleneck for PRMs is the acquisition of high-quality labeled data. To this end, early efforts employed human annotators guided by detailed instructions and rigorous protocols to achieve reliable annotation quality 18 . Nonetheless, such human annotation is labor-intensive, costly, and difficult to scale, which has led to a growing interest in automated labeling approaches 27 ; 36 ; 22 . Figure 1: MCE method shows significant deviation compared to human-annotated ground truth, regardless of model capability and sampling budget. A representative method for automated labeling is Monte Carlo Estimation (MCE). It defines a step’s reward as the probability of reaching a correct final answer, which is estimated by using a generative policy to sample numerous reasoning paths from the current step 27 ; 34 ; 19 ; 22 . However, step correctness is an intrinsic property of the reasoning trajectory, and should be invariant to policy choice. his highlights a fundamental conflict between MCE and process rewards, leading to significant label noise 8 . As shown in Figure 1 , MCE method correlate poorly with human annotations, regardless of the sampling budget or model capability. To combat this, we first analyze the sources of MCE labeling errors, which we categorize as: 1) False Positives (FP) , where an incorrect step leads to a correct final answer due to the policy’s powerful self-correction capabilities; and 2) False Negatives (FN) , where a correct step leads to an incorrect final answer due to subsequent failures in the reasoning path (as seen in Figure 2 ). Motivated by these observations, we propose a two-stage approach to mitigate the impact of above noise on process reward modeling. First, during the data labeling phase, we design a reflection-aware label correction mechanism, which corrects overestimated rewards for steps involving self-correction, thereby addressing FPs. Second, during the model training phase, we propose a N oise- A ware I terative T raining framework. This framework allows the model effectively to combine the initial labels with its own confidence to correct the labels, thereby accurately judging the correctness of each reasoning step. We conduct extensive experiments across diverse datasets and model scales. Compared to PRMs trained under noisy supervision, our model achieves up to a 27% absolute improvement in average F1 for step-level correctness discrimination, while reaching comparable performance to strong critic models such as GPT-4o. Under test-time scaling with Best-of- N , our PRM further improves final answer accuracy by 3.2–5.6% over baselines. Additional experiments show consistent gains as training data scales, indicating strong robustness and favorable scaling behavior. 2 Related Work 2.1 From Outcomes to Processes Process Reward Models (PRMs) provide fine-grained supervision over intermediate reasoning steps and have been shown to improve reasoning accuracy 26 ; 18 . Most existing PRMs model step correctness via independent classification with cross-entropy loss 3 ; 11 ; 14 ; 15 ; 31 , which ignores inter-step dependencies and is sensitive to noisy supervision. To better capture sequential structure, 17 propose Process Q-value Models (PQM) that optimize relative Q-value rankings. Beyond discriminative formulations, Generative PRMs assign rewards conditioned on explicitly generated chain-of-thoughts, enabling stronger generalization and test-time scaling 30 ; 35 ; 32 ; 33 . In contrast, our work focuses on improving the robustness of discriminative PRMs under noisy process supervision. 2.2 Data Annotation for Process Supervision Effective PRM training depends on high-quality process annotations, which are costly to obtain at scale. To reduce annotation overhead, prior work explores automated labeling via LLM-based annotators 24 ; 1 ; 28 or Monte Carlo Estimation (MCE), which estimates step correctness by sampling future reasoning trajectories 27 ; 22 ; 19 . However, MCE-based labels are highly sensitive to the sampling policy and often exhibit substantial noise. The most closely related work is 8 , which reduces label noise via self-denoising across rollouts of a fixed policy, our approach directly addresses this limitation by explicitly correcting reflection-induced overestimation and iteratively refining labels beyond a fixed policy. 3 Preliminary Figure 2: Illustration of MCE label noise. (Left) False Positive: A high reward is assigned to an incorrect step due to the policy’s subsequent self-correction. (Right) False Negative: A low reward is assigned to a correct step due to a subsequent policy failure. We first review Monte Carlo Estimation (MCE) for process supervision, then formalize discriminative process reward modeling, and finally analyze how MCE induces policy-dependent label noise. Figure 3: Overview of the proposed framework for robust process reward modeling. The framework consists of two stages. (Top right) In the data annotation stage, we propose a reflection-aware label correction mechanism, where an LLM judge detects reflection and self-correction behaviors within reasoning trajectories to identify and suppress overestimated rewards. (Bottom) In the training stage, we introduce a noise-aware iterative training framework that trains the process reward model with noise-robust objectives and progressively refines noisy labels based on model confidence across multiple stages. 3.1 Monte Carlo Estimation Given a generative policy \pi , MCE samples K future continuations from step s_{t} , producing trajectories \{\tau^{(k)}\}_{k=1}^{K} , and compares their final answers \mathrm{a}(\tau^{(k)}) with the ground-truth answer a^{*} . Two common estimation strategies are used: \displaystyle\hat{y}_{t}^{\text{SE}}(\pi) \displaystyle=\frac{1}{K}\sum_{k=1}^{K}\mathbbm{1}\bigl(\mathrm{a}(\tau^{(k)})=a^{*}\bigr),