ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

Dawei Li, Yuguang Yao, Zhen Tan, Huan Liu, Ruocheng Guo · 2026

Read ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents by Dawei Li, Yuguang Yao, Zhen Tan, Huan Liu, Ruocheng Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12294.txt.

Introduction

Dawei Li, Yuguang Yao, Zhen Tan, Huan Liu, and Ruocheng Guo build a test set for process reward models in tool-using agents. Reward-guided search lifts tool agents by steering their sampling over complex act spaces. Process reward models give step-level scores for that steering.

No firm test set yet exists for such models in tool settings. Their bench, named in the work, fills that gap. It also tests ways to advance the models. Sound step scores are key to agents that plan in many steps. Step scores guide search in long tool use chains.

Catalog record

CREATORDawei Li, Yuguang Yao, Zhen Tan, Huan Liu, Ruocheng Guo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-12294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents Dawei Li, Yuguang Yao, Zhen Tan, Huan Liu, Ruocheng Guo Abstract Reward-guided search methods have demonstrated strong potential in enhancing tool-using agents by effectively guiding sampling and exploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitoring. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we introduce ToolPRMBench, a large-scale benchmark specifically designed to evaluate PRMs for tool-using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interaction history, a correct action, a plausible but incorrect alternative, and relevant tool metadata. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verification pipeline is proposed to reduce label noise and ensure data quality. We conduct extensive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using. Code and data will be released at https://github.com/David-Li0406/ToolPRMBench . 1 Introduction In recent years, Large Language Models (LLMs) have achieved remarkable success, exhibiting exceptional performance across a wide range of applications—from complex reasoning and multilingualism to highly specialized fields ( OpenAI et al. 2024 ; Li et al. 2025b ; Ren et al. 2025 ; Yang et al. 2025b ; Tan et al. 2024b ; Wang et al. 2024c ; Zhang et al. 2025a ; Jiang and Ferraro 2024 ; Zhang et al. 2025c ; Qin et al. 2025 ; Zhang et al. 2025b ; Jiang et al. 2025 ; Qwen et al. 2025 ) . Complementing this progress, a rapidly growing area of focus is the deployment of LLMs as tool-using agents, which interact with external tools such as APIs, databases, and execution environments to solve complex, multi-step tasks Shen 2024 ; Huang et al. 2024 ; Tan et al. 2024a ; Lee et al. 2025 ; Zhao et al. 2025a ; Yu et al. 2025 ; Xu et al. 2025 ; Chang et al. 2025 ; Tan et al. 2025 . This paradigm significantly extends the capability of LLMs beyond pure text generation. However, effective tool-using remains challenging. As highlighted in recent work, even strong models frequently fail due to early mistakes that propagate through long sequences, while final outcome-based evaluation provides limited insight into where the reasoning process goes wrong Yao et al. 2022 ; Liang et al. 2025 ; Chae et al. 2025 ; Shahroz et al. 2025 . Benchmark Step-level Agent Tool-using Agent-RewardBench Men et al. 2025 ✗ ✓ Web-only AgentRewardBench Lù et al. 2025 ✗ ✓ Web-only WebRewardBench Chae et al. 2025 ✓ ✓ Web-only PRMBench Song et al. 2025 ✓ ✗ ✗ ToolPRMBench ✓ ✓ Diverse APIs Table 1: ToolPRMBench is the first benchmark that supports step-level evaluation for interactive agents with diverse tool APIs. Motivated by these challenges, recent research has increasingly explored reward-guided search as a way to improve tool-using agents. Instead of committing to a single trajectory, reward-guided methods perform searching or sampling over multiple candidate actions or plans. In tool-using and web agent settings, methods such as best-of- n or Monte Carlo tree search have shown promising results Chae et al. 2025 ; Agarwal et al. 2025 ; Zhang et al. 2025d . A central component of these approaches is a Process Reward Model (PRM) Snell et al. 2024 ; Zhao et al. 2025b , which provides step-level feedback to guide exploration and prune incorrect trajectories early. Compared to outcome-only rewards, PRMs offer finer-grained signals that are better aligned with the long-horizon tool-using tasks. However, despite their growing importance, evaluating PRMs for tool-using remains a challenging task. Tool-using, as an agent task, is characterized by long interactions and a large, structured action space, where errors can emerge at many intermediate steps and propagate over time; as a result, existing PRM benchmarks designed for general reasoning Song et al. 2025 or web agents Men et al. 2025 ; Lù et al. 2025 may not be directly applicable or sufficiently effective in tool-using scenarios. Moreover, existing PRM designs vary widely, ranging from LLM-as-a-judge methods Li et al. 2025a to general PRMs Yang et al. 2024 or agent- Chae et al. 2025 and tool-specialized PRMs. However, there is no unified benchmark to systematically evaluate their effectiveness in tool-using settings. To address this gap, we introduce ToolPRMBench , a large-scale benchmark designed specifically for evaluating process reward models for tool-using agents (Figure 1 ). ToolPRMBench is constructed on top of several representative tool-using benchmarks, covering diverse environments such as information-seeking, multi-step reasoning, and interactive tool execution. Each sample in ToolPRMBench consists of the interaction history, a correct action, an incorrect but plausible alternative, and associated tool metadata. To build the dataset, we combine offline sampling, which isolates local single-step errors around golden trajectories, and online sampling, which captures realistic multi-step failures from full agent rollouts. These candidate samples are further verified through a multi-LLM filtering pipeline to ensure label reliability. This construction process results in a diverse and challenging benchmark that enables fine-grained evaluation of PRMs at the decision-step level. Through extensive experiments on ToolPRMBench across a total of 17 large language models, general PRMs, and tool-specialized PRMs, we observe clear and consistent performance differences across model types. Our results further show that scaling model size and general capabilities benefit tool process reward modeling, while reinforcement learning demonstrates strong potential for improving robustness and generalization in tool-using PRMs. In addition, we conduct a series of analyses on ToolPRMBench, including meta-evaluation, data synthesis, cost analysis, and case studies, providing further insights to guide future research on reward modeling for tool-using agents. In summary, the contribution in this work is threefold: • First, we propose ToolPRMBench, a large-scale benchmark carefully constructed for systematic evaluation of PRMs in tool-using settings. • Second, we benchmark a series of LLMs, general PRMs, and tool-using specialized PRMs in ToolPRMBench, building a comprehensive PRMs leaderboard in the tool-using scenario. • Finally, we conduct further analysis, including meta-evaluation and cost analysis, providing findings and insights for future reward-guided trajectory searching in tool-using. 2 Related Work 2.1 Tool-using Benchmarks Tool-using agents extend LLMs by enabling them to interact with external tools or APIs rather than only producing text. This capability is necessary because many real-world tasks cannot be solved by language generation alone and require actual execution of external actions to achieve correct outcomes. Several benchmarks have been proposed to evaluate tool-using capabilities Farn and Shin 2023 ; Grattafiori et al. 2024 ; Wang et al. 2024a ; Lu et al. 2025 . Some focus on whether a model can utilize tools where appropriate Huang et al. 2024 ; Ning et al. 2024 , while others measure multi-hop or structured tool invocation, such as ToolHop Ye et al. 2025 and MCP-RADAR Gao et al. 2025 . Additionally, evaluation frameworks like T-Eval and Trajectory-Bench decompose tool utilization into sub-processes (instruction following, planning, reasoning, retrieval, etc.) to provide fine-grained analysis beyond end-to-end task success Chen

Read the complete text