OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

Ziyou Hu, Zhengliang Shi, Minghang Zhu, Haitao Li, Teng Sun, Pengjie Ren, Suzan Verberne, Zhaochun Ren · 2025

Read OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning by Ziyou Hu, Zhengliang Shi, Minghang Zhu, Haitao Li, Teng Sun, Pengjie Ren, Suzan Verberne, Zhaochun Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24636.txt.

Introduction

Ziyou Hu and seven co-authors present a new judge model for long, open replies from AI tools. Old judge models rely on what they know inside, so they miss faults that need proof from the world. Their model, OpenRM, can call tools to fetch proof while it grades.

The team trained it on more than twenty seven thousand pairs of replies made in a set way. Training scored both tool use and the final grade, so the model learned to judge from proof. On five test sets, OpenRM beat past judge models by a clear gap. It can also help steer a model as it writes.

Catalog record

CREATORZiyou Hu, Zhengliang Shi, Minghang Zhu, Haitao Li, Teng Sun, Pengjie Ren, Suzan Verberne, Zhaochun Ren
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24636

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning Ziyou Hu, Zhengliang Shi, Minghang Zhu, Haitao Li, Teng Sun, Pengjie Ren, Suzan Verberne, Zhaochun Ren Abstract Reward models (RMs) have become essential for aligning large language models (LLMs), serving as scalable proxies for human evaluation in both training and inference. However, existing RMs struggle on knowledge-intensive and long-form tasks, where evaluating correctness requires grounding beyond the model’s internal knowledge. This limitation hinders them from reliably discriminating subtle quality differences, especially when external evidence is necessary. To address this, we introduce OpenRM , a tool-augmented long-form reward model that systematically judges open-ended responses by invoking external tools to gather relevant evidence. We train OpenRM with Group Relative Policy Optimization (GRPO) on over 27K synthesized pairwise examples generated through a controllable data synthesis framework. The training objective jointly supervises intermediate tool usage and final outcome accuracy, incentivizing our reward model to learn effective evidence-based judgment strategies. Extensive experiments on three newly-collected datasets and two widely-used benchmarks demonstrate that OpenRM substantially outperforms existing reward modeling approaches. As a further step, we integrate OpenRM into both inference-time response selection and training-time data selection. This yields consistent gains in downstream LLM alignment tasks, highlighting the potential of tool-augmented reward models for scaling reliable long-form evaluation. Our code is available on OpenRM . 1 Introduction Reward models (RMs) have emerged as scalable and effective substitutes for human evaluators, playing a central role in aligning Large Language Models (LLMs) during both training and inference ( Guo et al. 2025 ; Chen et al. 2025b ) . By learning to predict human preferences over model outputs, RMs provide reliable supervision signals that guide reinforcement learning and support preference optimization at inference time ( Ouyang et al. 2022 ; Song et al. 2024 ; Zuo et al. 2025 ) . Benefiting from the rich world knowledge embedded in LLMs, RMs exhibit a certain degree of generalization capability, enabling them to perform effectively across diverse scenarios. As a result, building accurate, robust, and well-aligned reward models has become a critical step toward the development of safer and more useful intelligent systems. Despite the success, existing RMs still faces significant challenges when evaluating the knowledge-intensive long-form outputs generated by Deep Researcher systems ( Zheng et al. 2025 ; Zhu et al. 2025 ) . These texts typically require integrating various external information sources, which goes beyond the model’s internal knowledge. For example, evaluating the novelty of a research paper often depends on cross-referencing external scientific corpora (e.g., arXiv) ( Zhu et al. 2025 ) , while comparing travel itineraries drafted by downstream LLM applications demands up-to-date knowledge of specific destinations ( Chen et al. 2024 ) . Existing studies either rely on prompting commercial LLMs (e.g., GPT-4) to perform fact-checking via search engines ( Wei et al. 2024 ) , or use simple tool interactions for short-form factual verification (e.g., date checking via a calendar) ( Li et al. 2023 ) . However, these approaches exhibit limited flexibility and scalability, and their effectiveness in long-form reward modeling remains unclear. These limitations highlight a key open question: How can we design reward models that effectively evaluate long-form outputs in open-domain settings? To address the above question, we propose empowering reward models with the ability to use external tools during evaluation . Similar to task-solving agents ( Jin et al. 2025 ; Wang et al. 2024a ) , tool-augmented reward models can retrieve and reason over external information sources in real time, allowing them to make more accurate and context-aware judgments. Building on this intuition, we introduce OpenRM , a reward modeling framework designed to assess complex, knowledge-intensive long-form responses with the assistance of external tools. OpenRM explores how tool-use capabilities can be flexibly integrated into the reward modeling pipeline via reinforcement learning (RL), enabling the model to actively retrieve, verify, and reason over external information when forming evaluation judgments. Given two candidate responses, OpenRM first plans and executes a sequence of tool calls to retrieve supporting evidence, then verifies each response against this evidence, and finally selects the better answer. We train both the stepwise reasoning and the final judgment of the reward model via reinforcement learning using Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) . The objective combines (i) an intermediate tool‑use penalty that discourages irrelevant or excessive calls while encouraging task‑appropriate tool selection, and (ii) a terminal outcome‑accuracy reward that incentivizes reliably comparing the two responses and choosing the higher‑quality one. A key obstacle in training long-form reward models is the lack of reliable pairwise response data from real-world sources to serve as training data, as demonstrated in Table 1 . To address this, we propose a simple yet controllable partial synthesis framework that automatically generates queries and constructs positive–negative pairs. Specifically, we first perform target-aware query generation, prompting a strong LLM (e.g., DeepSeek-V3) to formulate a self-contained query from a sampled document. Then, we obtain preference-ordered responses by prompting the LLM answer the query with and without access to the reference document, thereby yielding a positive and a negative response, respectively. This method enables scalable supervision without costly human annotations. As a further step, we explore the utility of OpenRM in scaling LLM alignment at both training and inference stages. At inference time, we show that OpenRM can select a better response among candidates, improving long-form answer quality. At training time, we employ OpenRM as a data selector, where we compare task-solving trajectories sampled from LLMs and filtering with OpenRM , thereby obtaining a higher-quality training dataset. Our experiments demonstrate that LLMs trained with OpenRM -selected data achieve significant performance gains compared with those trained on data filtered by existing RMs, highlighting the potential of tool-augmented reward modeling. Our contributions are summarized as follows: (i) We introduce OpenRM , a reward model that can judge the long-form answer with the assistance of external tools, enabling accurate rewarding of complex agentic tasks. (ii) We propose a simple but controllable pipeline for automatic reward data construction, collecting 30K+ high-quality pair-wise training data with binary label in an unsupervised manner. (iii) Extensive experiments on five benchmarks, showing that OpenRM outperforms existing baselines significantly. (v) We demonstrate that OpenRM not only improves inference-time response selection but also serves as an effective data selector for training, paving the way for future research on LLM alignment. 2 Related Work 2.1 Reward modeling Reward modeling serves as a crucial bridge connecting human intent with model behavior, and is a key component in the development of artificial intelligence. Existing research mainly explores two paradigms of reward modeling: (1) Scalar Reward Models (Scalar RMs): This approach typically involves training a sequence classifier on top of a frozen large language model (LLM), turning reward modeling into a supervised classification problem that predicts human preferences or rating labels ( Wang et al. 2024c ; Liu et al. 2024b ) . (2) Generative Reward Models (a.k.a., LLM-as-Judge): This paradigm retains the generative capabilities of LLMs, leveraging their powerful language understanding and free-form reasoning abilities to produce preference judgments for pairwise comparisons or natural language evaluations ( Li et al. 2024a ; Zheng et al. 2023 ; Gu et al. 2024 ) . Compared to traditional scalar

Read the complete text