Entropy-Regularized Process Reward Model
Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, Tong Zhang · 2024
Read Entropy-Regularized Process Reward Model by Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, Tong Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11006.txt.
Introduction
Hanning Zhang and eight co-authors improve how reward models grade a chatbot's reasoning steps, in this 2024 paper. Process reward models score each step of a solution rather than only the final answer, which steers models toward sound reasoning. Math remains a weak spot, with systematic slips mid-solution.
Their model adds entropy regularization, borrowed from a theory of cautious decision making: the trained policy should improve without drifting too far from where it started. From that theory they derive a way to build the ideal reward model out of samples from the starting policy itself. On two math benchmarks, GSM8K and MATH, their model beat earlier process reward models by one to three points when picking the best of many tries. Gains also held under reinforcement training. The authors say the regularizer is the active ingredient.
Catalog record
| CREATOR | Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, Tong Zhang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11006
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Entropy-Regularized Process Reward Model Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, Tong Zhang Abstract Large language models (LLMs) have shown promise in performing complex multi-step reasoning, yet they continue to struggle with mathematical reasoning, often making systematic errors. A promising solution is reinforcement learning (RL) guided by reward models, particularly those focusing on process rewards, which score each intermediate step rather than solely evaluating the final outcome. This approach is more effective at guiding policy models towards correct reasoning trajectories. In this work, we propose an entropy-regularized process reward model (ER-PRM) that integrates KL-regularized Markov Decision Processes (MDP) to balance policy optimization with the need to prevent the policy from shifting too far from its initial distribution. We derive a novel reward construction method based on the theoretical results. Our theoretical analysis shows that we could derive the optimal reward model from the initial policy sampling. Our empirical experiments on the MATH and GSM8K benchmarks demonstrate that ER-PRM consistently outperforms existing process reward models, achieving 1% improvement on GSM8K and 2-3% improvement on MATH under best-of-N evaluation, and more than 1% improvement under RLHF. These results highlight the efficacy of entropy-regularization in enhancing LLMs’ reasoning capabilities. Reviewed on OpenReview: https://openreview.net/forum?id=cSxDH7N3x9 1 Introduction Large language models (LLMs) have demonstrated remarkable performance across numerous tasks, particularly in the challenging area of mathematical reasoning ( OpenAI et al., 2024 ; Yang et al., 2024 ; Dubey et al., 2024 ) . A key factor behind these successes is the use of synthetic data, as explored in works such as Meta-MathQA ( Yu et al., 2023 ) , MAmmoTH ( Yue et al., 2023 ) , and Open-MathInstruct ( Toshniwal et al., 2024b ) . Open-source models fine-tuned on these synthetic datasets have shown significant improvements in test accuracy on standard mathematical reasoning benchmarks such as MATH ( Hendrycks et al., 2021 ) and GSM8K ( Cobbe et al., 2021 ) . Following supervised fine-tuning, reinforcement learning (RL) has received considerable attention as a technique for further improving models’ reasoning abilities by aligning the model’s behavior with human values and goals, especially after the release of OpenAI-o1 model 1 1 1 https://openai.com/index/introducing-openai-o1-preview/ . The RL methods used in the current literature on LLM reasoning can be broadly categorized into two approaches. The first category focuses on training-time alignment, where RL is used to fine-tune the model to maximize an RL metric evaluated by a reward model. Examples include PPO ( Schulman et al., 2017 ) , GRPO ( Shao et al., 2024 ) . More recently, scaling inference-time computation has also been shown to enhance model performance by using reward models to guide LLM decoding. Techniques such as best-of-n sampling ( Dong et al., 2023 ) and Monte Carlo Tree Search (MCTS) ( Xie et al., 2024 ) are representative examples of this approach. All these methods require an external reward model to provide a reward signal to update the model parameters or rank the candidate responses. Regardless of the specific RL method, the quality of the reward function remains the most crucial factor, as it sets the upper bound for the algorithm’s performance. Figure 1 : Illustration of ER-PRM, along with other baseline methods to construct the process reward data and outcome reward data. The key idea of ER-PRM is to calculate the process reward from the sampling trajectories under entropy-regularization. One distinct feature of complex mathematical reasoning is that models typically require multiple steps of reasoning before producing the final answer ( Wei et al., 2023 ) . Accordingly, there are generally two types of reward models in the mathematical domain: the outcome reward model (ORM) and the process reward model (PRM). ORM provides an overall score for the quality of the final answer, while PRM assigns a score to each step in the reasoning process, providing more fine-grained feedback. Existing works ( Wang et al., 2024 ; Lightman et al., 2023 ) show that PRM outperforms ORM by a considerable margin. For instance, in Lightman et al. (2023) , if we use the PRM to select the final output from the candidate responses, we can achieve a test accuracy of 78.2%, as compared to the ORM with 72.4% and majority voting of 69.6%. Furthermore, PRM is more interpretable because it mirrors human behavior when evaluating reasoning paths. The PRM model is able to guide the policy model’s decoding at each individual step or rerank candidate answers to enhance accuracy and consistency during generation. Inspired by the superior performance of PRM, researchers have spent great efforts recently in the training of PRM. To get the process annotation for the PRM, one can ask humans to label the data at each step ( Lightman et al., 2023 ) , which is extremely expensive and inefficient. Since then, researchers have explored alternative labeling techniques. One representative example is the automatic labeling technique introduced by Math-Shepherd ( Wang et al., 2024 ) . The main idea of Math-Shepherd is to interpret the score of each step as its potential to deduce the correct final answer. Then, we can sample multiple trajectories starting from the intermediate step, and use the ratio of correct trajectories as a proxy for this score. The Math-Shepherd has proved as an effective way to generate high-quality process reward scores and even outperform human-labeled data. Despite its effectiveness, Math-Shepherd considers a traditional Markov Decision Process (MDP) framework, which differs from the typical RL practice used with LLMs. Since the introduction of reinforcement learning from human feedback (RLHF), the standard approach has been using an entropy-regularized reward relative to a reference model (often the initial model) ( Ouyang et al., 2022 ; Bai et al., 2022 ) . This also applies to RL in reasoning tasks where we formulate the problem as an entropy-regularized MDP ( Shao et al., 2024 ; Xiong et al., 2024b ; Zhong et al., 2024b ) . The main goal here is to balance optimizing the reward and staying close to the original policy, which is crucial because we start with a well-trained LLM, and we do not want the model to deviate too far from this initial strong foundation policy. In recognition of this gap, in this paper, we formulate the multi-step mathematical reasoning task under the entropy-regularized MDP framework, derive the mathematical principles of the process reward construction, and propose practical algorithms. Our contributions are summarized as follows. \bullet We propose the entropy-regularized process reward, a novel method for labeling the process reward score with entropy regularization. \bullet Based on the labeled data, we train a new process reward model to improve the performance. \bullet On the MATH and GSM8K datasets, we achieve significant improvements using the best-of-N evaluation method. Furthermore, by employing the reward model with rejection sampling, we observe a substantial enhancement in the performance of the policy model. \bullet To further boost the process reward model research, we will release all the data, code, and checkpoints to the community. 2 Related Work Mathematical Reasoning and Synthetic Data Generation. Mathematical problem solving is crucial for evaluating LLMs’ reasoning abilities. A common approach is Chain-of-Thought (CoT) prompting, which guides LLMs to solve problems step-by-step ( Wei