Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models

Ilgee Hong, Changlong Yu, Liang Qiu, Weixiang Yan, Zhenghao Xu, Haoming Jiang, Qingru Zhang, Qin Lu, Xin Liu, Chao Zhang, Tuo Zhao · 2025

Read Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models by Ilgee Hong, Changlong Yu, Liang Qiu, Weixiang Yan, Zhenghao Xu, Haoming Jiang, Qingru Zhang, Qin Lu, Xin Liu, Chao Zhang, Tuo Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16265.txt.

Introduction

Reward models grade the answers of language models during training, and the grades steer what the model learns. Newer generative graders write out their reasoning before scoring, but that reasoning stays short and shallow. An eleven-author team presents Think-RM, a training recipe that teaches such a grader to think at length first, using warm-up training on long reasoning examples followed by rule-based reinforcement learning.

The grader's self-guided reasoning supports reflection, testing of rival readings, and exploring alternatives. On a standard benchmark it beat both classic reward models and short-reasoning graders by about 8 percent. The team also adds a training pipeline that uses pairwise preference rewards directly, skipping a conversion step older methods needed. Models trained that way ended with better final performance than those trained the usual way.

Catalog record

CREATORIlgee Hong, Changlong Yu, Liang Qiu, Weixiang Yan, Zhenghao Xu, Haoming Jiang, Qingru Zhang, Qin Lu, Xin Liu, Chao Zhang, Tuo Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models Ilgee Hong, Changlong Yu, Liang Qiu, Weixiang Yan, Zhenghao Xu, Haoming Jiang, Qingru Zhang, Qin Lu, Xin Liu, Chao Zhang, Tuo Zhao Abstract Reinforcement learning from human feedback (RLHF) has become a powerful post-training paradigm for aligning large language models with human preferences. A core challenge in RLHF is constructing accurate reward signals, where the conventional Bradley-Terry reward models (BT RMs) often suffer from sensitivity to data size and coverage, as well as vulnerability to reward hacking. Generative reward models (GenRMs) offer a more robust alternative by generating chain-of-thought (CoT) rationales followed by a final reward. However, existing GenRMs rely on shallow, vertically scaled reasoning, limiting their capacity to handle nuanced or complex (e.g., reasoning-intensive) tasks. Moreover, their pairwise preference outputs are incompatible with standard RLHF algorithms that require pointwise reward signals. In this work, we introduce Think-RM, a training framework that enables long-horizon reasoning in GenRMs by modeling an internal thinking process. Rather than producing structured, externally provided rationales, Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities such as self-reflection, hypothetical reasoning, and divergent reasoning. To elicit these reasoning abilities, we first warm-up the models by supervised fine-tuning (SFT) over long CoT data. We then further improve the model’s long-horizon abilities by rule-based reinforcement learning (RL). In addition, we propose a novel pairwise RLHF pipeline that directly optimizes policies using pairwise preference rewards, eliminating the need for pointwise reward conversion and enabling more effective use of Think-RM outputs. Experiments show that Think-RM achieves state-of-the-art results on RM-Bench, outperforming both BT RM and vertically scaled GenRM by 8%. When combined with our pairwise RLHF pipeline, it demonstrates superior end-policy performance compared to traditional approaches. This depth-oriented approach not only broadens the GenRM design space but also establishes a new paradigm for preference-based policy optimization in RLHF. The code, datasets, and models are publicly available at https://github.com/IlgeeHong/Think-RM . 1 Georgia Institute of Technology 2 Amazon 1 Introduction Reinforcement learning from human feedback (RLHF) has emerged as a powerful post-training paradigm for large language models (LLMs), enabling them to better follow instructions [ 1 , 2 , 3 , 4 ] , reason over multiple steps [ 5 , 6 , 7 , 8 ] , and comply with safety constraints [ 9 , 10 , 11 ] . By iteratively shaping the model with a learned reward signal aligned to human preferences, RLHF bridges the gap between pretraining objectives and real-world usage. A central challenge in RLHF lies in constructing an accurate reward signal. A conventional approach is to use a Bradley-Terry reward model (BT RM), which maps a prompt and response pair to a single scalar score by minimizing empirical risk over preference data [ 12 , 13 ] . While BT RM is straightforward to implement and easy to train, they often overfit to specific patterns in the training data and are highly sensitive to dataset size and coverage, frequently leading to reward over-optimization and reward hacking [ 14 , 15 , 16 , 17 , 18 ] . Generative reward models (GenRMs) offer a promising alternative to conventional discriminative approaches [ 18 , 19 , 20 , 21 , 22 ] . By training an LLM to generate a chain-of-thought (CoT) explanation followed by a final reward or preference, GenRM leverages the pretrained model’s existing knowledge and shows greater robustness to data scarcity and distribution shifts, demonstrating stronger out-of-distribution (OOD) performance [ 20 , 21 , 22 ] . Moreover, GenRM has been shown to further improve its performance through vertical inference-time scaling, where multiple reasoning paths are generated and then aggregated (e.g., by majority voting or averaging) to produce a more reliable reward or preference estimate [ 19 , 20 , 21 , 22 ] . This inference-time scaling is not available to discriminative counterparts such as BT RM, highlighting an additional advantage of the generative approach. (a) Self-Reflection (b) Hypothetical Reasoning (c) Divergent Reasoning Figure 1: Examples of advanced reasoning abilities enabled by Think-RM. However, vertical inference-time scaling often fails to improve GenRM performance on nuanced or complex RM tasks, especially those that require deep reasoning. While aggregating outputs from multiple reasoning paths improves self-consistency, each path generated by existing GenRM is typically shallow (limited to a few hundred tokens) making it difficult for any single path to fully capture complex or subtle implicit context. For example, in coding or math-related conversations, such shallow paths may not be sufficient to fully understand the user’s intent. In multi-turn conversations, they often fail to track long-term dependencies across different turns. In addition, shallow CoT reasoning is often insufficient to detect a single false statement embedded within an otherwise fluent and well-structured response. Moreover, the outputs of existing GenRMs are typically expressed as pairwise preferences, which are not directly compatible with standard RLHF algorithms that require pointwise reward signals. In contrast, scaling along the horizontal dimension, where the model reasons more extensively within a single trajectory, remains largely underexplored, despite its success in improving the quality of reasoning in other language model applications [ 23 , 24 , 25 ] . In this work, we introduce Think-RM , a new training framework that transforms a non-reasoning pretrained LLM (e.g., Llama series [ 26 , 27 ] ) into a GenRM equipped with long-horizon thinking capabilities by modeling an internal thinking process. Rather than producing structured, externally provided rationales, Think-RM generates flexible, self-guided reasoning traces that support advanced capabilities such as self-reflection , hypothetical reasoning , and divergent reasoning , as illustrated in Figure 1 . This enables the model to solve reasoning-heavy RM tasks by extending a single CoT trajectory from hundreds to thousands of tokens. To stimulate the reasoning abilities for the non-reasoning model, we first warm up the model with supervised fine-tuning (SFT). We generate multiple long CoT trajectories for each GenRM prompt using a pretrained reasoning model and select the longest correct one, which we use to fine-tune the model. After the SFT warm-up, we further refine the model’s overly long or noisy reasoning process using rule-based reinforcement learning. In addition, we propose a novel pairwise RLHF pipeline that directly optimizes policies using pairwise preference rewards, eliminating the need for pointwise reward conversion and enabling more effective use of Think-RM’s outputs. Extensive experiments show that Think-RM outperforms both the BT RM and the vertically scaled standard GenRM on both in-distribution (ID) and OOD tasks. In particular, it outperforms these baselines by 8% on RM-Bench [ 28 ] , a challenging benchmark requiring intensive reasoning, achieving state-of-the-art performance among all publicly available RMs under 10B using only 6K training data. Furthermore, integrating Think-RM into our pairwise RLHF pipeline yields stronger end-policy performance compared to traditional pointwise RLHF with BT RMs. By shifting the modeling paradigm from breadth to depth, Think-RM not only expands the design space of generative reward modeling, but also establishes a new foundation for preference-based policy optimization in RLHF, paving the way toward better alignment of LLMs with more complex objectives. 2 Related Work Bradley-Terry (BT) Reward Modeling. The BT framework [ 29 ] is a conventional approach to reward modeling in RLHF, where a discriminative model learns to predict scalar rewards from pairwise

Read the complete text