Process Reward Models That Think

Muhammad Khalifa ♡ , Rishabh Agarwal ♠ , Lajanugen Logeswaran ♢ , Jaekyeom Kim ♢ , Hao Peng ♣ , Moontae Lee ♢ , Honglak Lee ♡♢ , Lu Wang ♡ ♡ University of Michigan ♠ Mila ♢ LG AI Research ♣ University of Illinois Urbana-Champaign [email protected] · 2025

Read Process Reward Models That Think by Muhammad Khalifa ♡ , Rishabh Agarwal ♠ , Lajanugen Logeswaran ♢ , Jaekyeom Kim ♢ , Hao Peng ♣ , Moontae Lee ♢ , Honglak Lee ♡♢ , Lu Wang ♡ ♡ University of Michigan ♠ Mila ♢ LG AI Research ♣ University of Illinois Urbana-Champaign [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16828.txt.

Introduction

Reward models that grade each step of a solution power modern test-time scaling, but step-level labels are expensive. Muhammad Khalifa and co-authors build ThinkPRM in 2025, a verifier that thinks out loud, checking each step with its own chain of reasoning before scoring it. Fine-tuning leaned on the reasoning skill of long-thought models, using a hundredth of the labels rival graders needed.

ThinkPRM beat both judge-style models and label-hungry graders on math and reasoning benchmarks, and held its lead on out-of-domain science and code sets. Given equal token budgets, its verification scaled better than a judge's. Code and models are public.

Catalog record

CREATORMuhammad Khalifa ♡ , Rishabh Agarwal ♠ , Lajanugen Logeswaran ♢ , Jaekyeom Kim ♢ , Hao Peng ♣ , Moontae Lee ♢ , Honglak Lee ♡♢ , Lu Wang ♡ ♡ University of Michigan ♠ Mila ♢ LG AI Research ♣ University of Illinois Urbana-Champaign [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16828

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Process Reward Models That Think Muhammad Khalifa ♡ , Rishabh Agarwal ♠ , Lajanugen Logeswaran ♢ , Jaekyeom Kim ♢ , Hao Peng ♣ , Moontae Lee ♢ , Honglak Lee ♡♢ , Lu Wang ♡ ♡ University of Michigan ♠ Mila ♢ LG AI Research ♣ University of Illinois Urbana-Champaign [email protected] Abstract Step-by-step verifiers—also known as process reward models (PRMs)—are a key ingredient for test-time scaling. PRMs require step-level supervision, making them expensive to train. This work aims to build data-efficient PRMs as verbalized step-wise reward models that verify every step in the solution by generating a verification chain-of-thought (CoT). We propose ThinkPRM , a long CoT verifier fine-tuned on orders of magnitude fewer process labels than those required by discriminative PRMs. Our approach capitalizes on the inherent reasoning abilities of long CoT models, and outperforms LLM-as-a-Judge and discriminative verifiers—using only 1% of the process labels in PRM800K—across several challenging benchmarks. Specifically, ThinkPRM beats the baselines on ProcessBench, MATH-500, and AIME ’24 under best-of-N selection and reward-guided search. In an out-of-domain evaluation on a subset of GPQA-Diamond and LiveCodeBench, our PRM surpasses discriminative verifiers trained on the full PRM800K by 8% and 4.5%, respectively. Lastly, under the same token budget, ThinkPRM scales up verification compute more effectively compared to LLM-as-a-Judge, outperforming it by 7.2% on a subset of ProcessBench. Our work highlights the value of generative, long CoT PRMs that can scale test-time compute for verification while requiring minimal supervision for training. 1 1 1 Our code, data, and models will be released at https://github.com/mukhal/thinkprm . Figure 1: Left: Verifier F1-score on ProcessBench ( Zheng et al. 2024 ) . ThinkPRM -14B, trained on 8K process labels or 1K synthetic examples, outperforms discriminative PRMs trained on about 100x more data. Right: Accuracy over MATH-500 using Llama-3.2-3B-Instruct as generator with different verifiers. ThinkPRM -1.5B, trained using the same 8K labels, outperforms LLM-as-a-judge and discriminative verifiers in reward-guided search on MATH-500. The LLM-as-a-judge in both figures uses the same base model as ThinkPRM . 1 Introduction Reasoning with large language models (LLMs) can substantially benefit from utilizing more test compute ( Jaech et al. 2024 ; Guo et al. 2025 ; Akyürek et al. 2024 ) . This typically depends on a high-quality process reward model (PRM)—also known as a process verifier—to select promising paths for search or ranking ( Cobbe et al. 2021 ; Li et al. 2023 ; Wu et al. 2024 ; Brown et al. 2024 ) . Given a problem-solution pair, the PRM provides a score that indicates whether the solution or part of it is correct. To date, PRMs have mostly assumed the form of discriminative classifiers, trained to discern correct from incorrect reasoning ( Uesato et al. 2022 ; Zhang et al. 2025 ) . However, training discriminative PRMs requires access to process labels, i.e., step-level annotations, which either require extensive human annotation ( Lightman et al. 2023 ; Zheng et al. 2024 ) , gold step-by-step solutions ( Khalifa et al. 2023 ) , or compute-intensive rollouts ( Luo et al. 2024 ; Wang et al. 2023b ; Chen et al. 2024a ) . LLM-as-a-judge can enable generative verification by treating verification as text generation without any training ( Wang et al. 2023a ; Liu et al. 2023b ; Zheng et al. 2023 ) . Despite the advantages of LLM-as-a-judge such as data-efficiency and interpretability, it has been known to perform quite poorly compared to specialized reward models in the context of complex reasoning ( Lambert et al. 2024 ; Zhang et al. 2024b ; Chen et al. 2024c ) , as uncustomized LLMs frequently fail to recognize incorrect reasoning ( Huang et al. 2023 ; Zhang et al. 2024a ; Ye et al. 2024 ) . This raises the question: How can we maintain both the data-efficiency and interpretability advantages and the superior performance of discriminative PRMs? Figure 2: ThinkPRM supports scaling verification compute by thinking longer. We approach this question with the following insight: LLM-as-a-judge verification can greatly benefit from scaling up the verifier’s inference compute—specifically, by enabling it to think through a verification chain of thought (CoT). To this end, we repurpose open-weight large reasoning models (LRMs; Team 2024 ; Guo et al. 2025 ; Li et al. 2025 ) as the foundation for generative PRMs through lightweight tuning. Specifically, we train LRMs using synthetic data ( Kim et al. 2023 ; Wang et al. 2024 ) , utilizing as few as 8K process labels, and obtain ThinkPRM —a PRM that not only surpasses LLM-as-a-judge under the same base model, but also outperforms discriminative PRMs trained on two orders of magnitude more data across a variety of test-time scaling scenarios. To obtain ThinkPRM , we train four different reasoning models, namely R1-Distill-Qwen{1.5B,7B,14B} ( Guo et al. 2025 ) , and QwQ-32B-Preview ( Team 2024 ) and our training is extremely efficient, e.g., training our largest PRM takes only 4 hours on a single A100 80GB. We extensively evaluate ThinkPRM as a standalone verifier on ProcessBench ( Zheng et al. 2024 ) , under different Best-of-N selection and verifier-guided search scenarios—using both in- and out-of-domain tasks. ThinkPRM -14B, based on R1-Distill-Qwen-14B, outperforms a discriminative PRM based on the same base model in terms of accuracy while using far fewer supervision signals as in Figure 1 left. In addition, ThinkPRM -1.5B, based on R1-Distill-Qwen-1.5B, demonstrates strong performance on MATH-500 ( Hendrycks et al. 2021 ) under guided beam search, shown in Figure 1 right. Lastly, as shown in Figure 2 , ThinkPRM can effectively utilize more verification compute than LLM-as-a-judge, by forcing it to think for more tokens. All these results are obtained while training on 1K synthetic examples, utilizing only 8K process labels. Our work highlights the promise of long CoT verification that verifies reasoning with reasoning, and that can effectively scale up both generator and verifier compute. To summarize, this work makes the following major contributions: • We introduce a simple recipe to obtain ThinkPRM —a generative PRM that verifies solutions via long CoT reasoning. Concretely, ThinkPRM can be efficiently trained from off-the-shelf reasoning model via finetuning on synthetic data. • ThinkPRM uses only 8K process labels and outperforms discriminative PRMs, trained on two orders of magnitude more labels by up to 8% Macro F1 points on ProcessBench. • We further show that ThinkPRM outperforms strong baseline PRMs in best-of-N and guided-search setups, on two in-domain tasks: MATH-500, AIME 2024, and two out-of-domain tasks: GPQA-Diamond and LiveCodeBench. ThinkPRM also outperforms LLM-as-a-judge baselines, and can enable better scaling of verifier compute by thinking longer. 2 Background and Related Work Process reward models (PRMs) evaluate reasoning solutions step-by-step by taking a problem and a multi-step solution prefix as input, and producing a sequence of scores, each assessing the correctness or quality of a specific step. PRMs can be categorized as discriminative or generative. Discriminative PRMs. Discriminative PRMs are trained as classifiers that directly predict numerical correctness scores for each step, typically relying on extensive step-level annotations ( Uesato et al. 2022 ; Lightman et al. 2023 ; Zhang et al. 2025 ) . Given a solution prefix, discriminative PRMs encode the solution text and employ a classification head to produce step-level scores, usually

Read the complete text