Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs

Zhangyin Feng, Qianglong Chen, Ning Lu, Yongqian Li, Siqi Cheng, Shuangmu Peng, Duyu Tang, Shengcai Liu, Zhirui Zhang · 2025

Read Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs by Zhangyin Feng, Qianglong Chen, Ning Lu, Yongqian Li, Siqi Cheng, Shuangmu Peng, Duyu Tang, Shengcai Liu, Zhirui Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11227.txt.

Introduction

Feng and co-authors ask, in a 2025 machine learning paper, whether separate step-checking models are needed to train reasoning in large language models. A popular view holds that such checkers, called process reward models, are key to better reasoning. Recent results from pure trial-and-error training, with no checker, challenge that view.

The team finds that skill at solving problems and skill at judging steps grow together during pure training. Current checkers even lose to a simple majority vote when used with top models. Their fix lets a model grade and rerank its own answers, which helps as more answers are sampled. Still, the self-grading is often wrong on hard problems, so more training scale is needed.

Catalog record

CREATORZhangyin Feng, Qianglong Chen, Ning Lu, Yongqian Li, Siqi Cheng, Shuangmu Peng, Duyu Tang, Shengcai Liu, Zhirui Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11227

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs Zhangyin Feng, Qianglong Chen, Ning Lu, Yongqian Li, Siqi Cheng, Shuangmu Peng, Duyu Tang, Shengcai Liu, Zhirui Zhang Abstract The development of reasoning capabilities represents a critical frontier in large language models (LLMs) research, where reinforcement learning (RL) and process reward models (PRMs) have emerged as predominant methodological frameworks. Contrary to conventional wisdom, empirical evidence from DeepSeek-R1 demonstrates that pure RL training focused on mathematical problem-solving can progressively enhance reasoning abilities without PRM integration, challenging the perceived necessity of process supervision. In this study, we conduct a systematic investigation of the relationship between RL training and PRM capabilities. Our findings demonstrate that problem-solving proficiency and process supervision capabilities represent complementary dimensions of reasoning that co-evolve synergistically during pure RL training. In particular, current PRMs underperform simple baselines like majority voting when applied to state-of-the-art models such as DeepSeek-R1 and QwQ-32B. To address this limitation, we propose Self-PRM, an introspective framework in which models autonomously evaluate and rerank their generated solutions through self-reward mechanisms. Although Self-PRM consistently improves the accuracy of the benchmark (particularly with larger sample sizes), analysis exposes persistent challenges: The approach exhibits low precision (<10%) on difficult problems, frequently misclassifying flawed solutions as valid. These analyses underscore the need for continued RL scaling to improve reward alignment and introspective accuracy. Overall, our findings suggest that PRM may not be essential for enhancing complex reasoning, as pure RL not only improves problem-solving skills but also inherently fosters robust PRM capabilities. We hope these findings provide actionable insights for building more reliable and self-aware complex reasoning models. 1 Introduction Recent advances in reinforcement learning (RL) scaling have substantially improved the ability of large language models (LLMs) to perform complex reasoning tasks. This progress is exemplified by the emergence of state-of-the-art (SOTA) models such as Kimi 1.5 team2025kimi , OpenReasonerZero OpenReasonerZero2025 , DeepSeek-R1 guo2025deepseek , QwQ-32B qwen2 ; qwq-32b-preview , and Claude-3.7-Sonnet Claude , all of which demonstrate remarkable problem-solving capabilities achieved through pure RL training. Alongside this RL-driven progress, process reward models (PRMs) liu2025can ; wu2024inference ; NEURIPS2024_30dfe47a have emerged as an alternative paradigm for reasoning supervision, with the aim of evaluating and optimizing the quality of intermediate reasoning steps. Despite their conceptual appeal, PRMs encounter three fundamental limitations: (1) the inherent ambiguity in defining granular reasoning steps, (2) prohibitive annotation costs, and (3) systemic vulnerability to reward hacking mechanisms. In particular, the DeepSeek-R1 technical report highlights these limitations guo2025deepseek , demonstrating that PRMs did not yield significant gains in the performance of complex reasoning. However, recent work continues to explore the potential of PRMs. For example, PRIME cui2025process introduces implicit reward mechanisms to bypass the need for labeled steps, GenPRM zhao2025genprm uses code verification and relative progress estimation to improve PRM performance, and Qwen ProcessBench zheng2024processbench offers a benchmark to assess the quality of the reasoning trace. These efforts underscore the ongoing interest in PRMs as a path to better reasoning supervision. However, a crucial question remains largely unexplored: Is explicit PRM training truly necessary for developing models capable of process-level reasoning? In other words, can RL training alone – in the absence of any process supervision – lead to models that are both strong problem solvers and capable evaluators of reasoning quality? To answer this, in this work, we present the first systematic empirical study exploring the intrinsic connection between RL training and PRM capabilities. Our key insight is that problem-solving and process supervision are not orthogonal skills, but rather complementary competencies that co-evolve during pure RL training. Through a series of experiments, we examine how reasoning models trained solely with rule-based rewards develop strong process-level judgement capabilities, without access to fine-grained supervision. Through a series of controlled experiments on math reasoning tasks, we demonstrate that RL-trained models like DeepSeek-R1 and QwQ-32B exhibit strong process judgement abilities, exceeding those of models explicitly trained with PRMs. In addition, we find that existing PRMs provide little to no benefit when applied to these models for reranking, often underperforming compared to simple heuristics such as majority voting. To address this, we introduce Self-PRM, an introspective approach where a model leverages its own internal reward signal to re-rank sampled outputs. Self-PRM consistently improves performance – particularly at higher sample sizes – by aligning candidate selection with the model’s own reasoning preferences. Despite these improvements, a detailed analysis reveals that Self-PRM suffers from low precision on challenging problems, often misclassifying incorrect solutions as correct. Stronger models, such as DeepSeek-R1 and QwQ-32B, exhibit more reliable self-evaluation, but the issue persists, highlighting the limitations of introspective scoring when reward alignment is imperfect. These findings challenge conventional assumptions about the necessity of process supervision and suggest that lightweight, introspective training signals – whether through self-evaluation or continued RL scaling – may provide a viable path forward. In summary, our main contributions are as follows: • We provide the first systematic study demonstrating that RL training alone – without any process-level supervision – can endow models with strong PRM capabilities. • We empirically show that existing PRMs fail to improve performance when applied to strong RL-trained models, and often underperform simple baselines such as majority voting. • We introduce the Self-PRM framework, in which models rerank their own outputs using internal reward signals, achieving consistent performance gains – while also revealing limitations in precision that highlight the need for improved reward alignment. 2 Preliminaries Reinforcement Learning. RL provides a framework in which an agent learns to make sequential decisions by interacting with an environment to maximize cumulative rewards. Formally, RL is typically modeled as a Markov Decision Process (MDP), defined by the tuple (\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma) , where \mathcal{S} is the state space, \mathcal{A} the action space, \mathcal{P} the state transition dynamics, \mathcal{R} the reward function, and \gamma the discount factor. In the context of LLMs, RL is widely used to align model behavior with human preferences. One prominent approach is Reinforcement Learning from Human Feedback (RLHF) Ouyang2022TrainingLM , where a reward model (RM) is trained to approximate human judgements over generated outputs. The base language model is then optimized using RL to maximize the reward provided by this RM. Several methods have been proposed to implement this optimization process effectively, including PPO Schulman2017ProximalPO , DPO Rafailov2023DirectPO , GRPO Shao2024DeepSeekMathPT , DAPO Yu2025DAPOAO and VAPO yuan2025vapo etc. These methods differ in how explicitly they use reward signals, but all share the goal of training LLMs to produce outputs that are more helpful, truthful, and aligned. Notably, recent RL-based models – including DeepSeek-R1 guo2025deepseek , Kimi1.5 team2025kimi , QwQ-32B qwq-32b-preview , OpenReasonerZero OpenReasonerZero2025 – achieve state-of-the-art performance on complex reasoning tasks without process-level supervision. Meanwhile, some works gou2024tora ; jin2025search ; song2025r1 ; li2025torlscalingtoolintegratedrl ; feng2025retoolreinforcementlearningstrategic are exploring RL with tools for efficient reasoning. While Retool feng2025retoolreinforcementlearningstrategic use DAPO-MATH-17k and DAPO for RL training to guide LLMs towards optimal strategies for leveraging external computational tools during reasoning, ToRL li2025torlscalingtoolintegratedrl leverage 7B-base to improve the complex reasoning ability. Process Reward Models. PRMs NEURIPS2024_30dfe47a ; Lightman2023LetsVS ; liu2025can ; ma2023let ; wang2025visualprm ; wu2024inference ; Zhong2025DyveTF ; uesato2022solving ; li2025process are designed to evaluate the reasoning process of LLMs, not just the final answer. They provide rewards or scores for each

Read the complete text