What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-boosted Mathematical Reasoning

Yiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu, Weiqi Luo · 2024

Read What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-boosted Mathematical Reasoning by Yiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu, Weiqi Luo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15904.txt.

Introduction

Yiran Ma and co-authors probe what step-level reward models truly reward in math reasoning. Their 2024 paper studies models that grade each step of a solution, guiding search toward sound proofs. Tree search methods can label step quality on their own, and the combo works well. Yet why it works stayed murky.

Stripping the plain-language thought text from steps barely hurt the graders. The models judge the logical weave of math notation keenly, but stumble on the same logic in words. So their strength leans on formal structure more than on read sense. The authors say future graders can drop bulk text and focus on the math core. That shift could make process supervision cheaper and sharper.

Catalog record

CREATORYiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu, Weiqi Luo
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15904

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-boosted Mathematical Reasoning Yiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu, Weiqi Luo Abstract Step-level reward models (SRMs) can significantly enhance mathematical reasoning performance through process supervision or step-level preference alignment based on reinforcement learning. The performance of SRMs is pivotal, as they serve as critical guidelines, ensuring that each step in the reasoning process is aligned with desired outcomes. Recently, AlphaZero-like methods, where Monte Carlo Tree Search (MCTS) is employed for automatic step-level preference annotation, have proven particularly effective. However, the precise mechanisms behind the success of SRMs remain largely unexplored. To address this gap, this study delves into the counterintuitive aspects of SRMs, particularly focusing on MCTS-based approaches. Our findings reveal that the removal of natural language descriptions of thought processes has minimal impact on the efficacy of SRMs. Furthermore, we demonstrate that SRMs are adept at assessing the complex logical coherence present in mathematical language while having difficulty in natural language. These insights provide a nuanced understanding of the core elements that drive effective step-level reward modeling in mathematical reasoning. By shedding light on these mechanisms, this study offers valuable guidance for developing more efficient and streamlined SRMs, which can be achieved by focusing on the crucial parts of mathematical reasoning. Introduction Figure 1: Each step in an LLM’s process of solving mathematical problems can be divided into the thought process and the execution of corresponding calculations. We find that natural language descriptions of the thought processes are not essential for step-level reward modeling. Large Language Models (LLMs) have demonstrated their remarkable capabilities across a wide range of tasks, such as information extraction, natural language understanding, etc ( Zhao et al. 2023 ) , totally revolutionizing the deep learning community. Among these capabilities, reasoning stands out as a critical area of focus, especially mathematical reasoning, which needs to be further improved due to its complex nature. Numerous studies have shown that multi-step reasoning often facilitated through Chain-of-Thought (CoT) prompting, can significantly enhance model performance on reasoning tasks ( Zhou et al. 2023 ; Besta et al. 2024 ; Ding et al. 2023 ; Yao et al. 2024 ; Wang et al. 2022 ; Wei et al. 2022 ; Zheng et al. 2024 ; Li et al. 2024 ; Zhan et al. 2024 ) . Recently, guided tree-search methods further improved reasoning performance by exploring various reasoning paths through online simulation to identify the optimal solution paths ( Hao et al. 2023 ; Hao et al. 2024 ; Feng et al. 2023 ) . Although a better reasoning path leads to a better performance, the length of these reasoning chains leads to an exponential increase in the search space, resulting in substantial computational costs. Given the high expense of LLM inference, performing an online tree search for each reasoning problem introduces repeated and unnecessary overhead. To address this issue, step-level reward models (SRM) was proposed to improve search efficiency. Lightman et al. 2023 introduced the process reward model (PRM), which employs human-annotated step-level scores for reward modeling, and Ma et al. 2023 further demonstrated the effectiveness of SRMs in math reasoning and coding tasks. Then, Math-Shepherd ( Wang et al. 2024 ) , systematically generates step-level preference data through exhaustive reasoning process traversal to train reward models and reinforce the model’s capabilities. More recently, inspired by AlphaZero, Monte Carlo Tree Search (MCTS) ( Xie et al. 2024 ; Chen et al. 2024a ; Chen et al. 2024b ) was then used for collecting preferences more efficiently because of its capability of balancing exploration and exploitation. These trained SRMs can effectively enhance reasoning performance by either assisting step-level preference alignment with proximal policy optimization (PPO) during training stage or serving as step verifiers during inference stage. Despite the significant achievements in mathematical reasoning performance achieved by the SRMs constructed by MCTS-based method, the exact workings of these reward models and what they are truly rewarding remain unclear. Brain and cognitive scientists have argued that diverse thinking and reasoning processes do not necessarily rely on natural language. ( Fedorenko, Piantadosi, and Gibson 2024 ) . A skilled human mathematician, for instance, can determine whether a mathematical expression is logically coherent and numerically correct without the participation of the natural language. Building on this idea, our research explores a similar hypothesis for LLMs: that natural language descriptions of thought processes are not essential for mathematical reasoning within these models. We suppose that LLMs can be trained to recognize preferences for mathematical language directly during problem-solving, without relying on natural language descriptions. This implies that LLMs might be capable of understanding and processing mathematical reasoning through the intrinsic structure of mathematical language, potentially leading to more efficient and focused training methods that bypass the need for natural language explanations. Furthermore, it is believed that incorrect solutions often arise from wrong mathematical calculations or logical errors ( Zhang et al. 2024 ) , with the latter being more challenging ( Chen et al. 2024a ) . Therefore, we further investigate the effectiveness of SRMs in evaluating logical coherence in pure mathematical language, demonstrating that the improvements are not merely the result of encouraging correct calculations within a single step. Additionally, and somewhat surprisingly, we found that SRMs struggle to learn how to evaluate logical coherence in natural language. This will further support that natural language is not necessary for step-level reward modeling. Figure 2: Illustration of the role of SRMs in mathematical reasoning and the SRMs with different input structures we investigate. To investigate the respective roles of natural language and mathematical language in step-level reward modeling, we decompose each step of the reasoning path into two components: natural language descriptions of thought processes and math expressions ( Figure 1 ). The ablation studies are conducted by selectively removing different parts from the inputs of the SRMs. This decomposition mirrors the human problem-solving process in mathematics, which typically involves an initial phase of thinking through the problem, followed by the execution of calculations based on that thought process. The thought processes include the strategy to be taken in that step, while the calculations are the executions of the thought processes. In other words, our decomposition aims to separate the natural language (composing the ‘thoughts’) from the mathematical expressions (contained in the execution of ‘thoughts’). This framework aims to foster a deeper understanding of the role of natural language for step-level reward modeling. To summarize, our experiments support that SRMs appear to have some intrinsic affinity for mathematical expression, not natural language. Specifically, we propose the following key insights. 1. Natural language descriptions of thought processes are not necessary for successful step-level reward modeling. 2. SRMs not only promote accurate calculations within individual steps but also effectively assess the challenging logical coherence in mathematical language. 3. Assessing logical coherence in natural language is difficult, and SRMs often struggle with this task. Preliminaries Markov Decision Process Definition A Markov Decision Process (MDP) is a mathematical framework used to model decision-making problems. This framework is fundamental for addressing a wide range of reinforcement learning (RL) problems where the outcomes are partially random and partially controllable. An MDP is defined by a tuple (S,A,P,R,\gamma)

Read the complete text