VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models
Yuchen Yan, Jin Jiang, Zhenbang Ren, Yijun Li, Xudong Cai, Yang Liu, Xin Xu, Mengdi Zhang, Jian Shao, Yongliang Shen, Jun Xiao, Yueting Zhuang · 2025
Read VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models by Yuchen Yan, Jin Jiang, Zhenbang Ren, Yijun Li, Xudong Cai, Yang Liu, Xin Xu, Mengdi Zhang, Jian Shao, Yongliang Shen, Jun Xiao, Yueting Zhuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15801.txt.
Introduction
Yuchen Yan and co-authors introduce VerifyBench in 2025, a test set for the checkers that grade reasoning models during training. Such models learn by having their answers verified against reference answers. Existing benchmarks mostly compare pairs of answers instead, leaving verification itself untested. The team built two benchmarks, one standard and one hard, with careful human labeling.
Larger checker models did well on plain cases. On hard cases, every system showed much room to improve. The authors sort errors by task and type to show where checkers fail. Better verification would lift the reasoning models trained with it, they argue. The benchmarks give the field a shared yardstick.
Catalog record
| CREATOR | Yuchen Yan, Jin Jiang, Zhenbang Ren, Yijun Li, Xudong Cai, Yang Liu, Xin Xu, Mengdi Zhang, Jian Shao, Yongliang Shen, Jun Xiao, Yueting Zhuang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models Yuchen Yan, Jin Jiang, Zhenbang Ren, Yijun Li, Xudong Cai, Yang Liu, Xin Xu, Mengdi Zhang, Jian Shao, Yongliang Shen, Jun Xiao, Yueting Zhuang Abstract Large reasoning models such as OpenAI o1 and DeepSeek-R1 have demonstrated remarkable performance in complex reasoning tasks. A critical component of their training is the incorporation of reference-based reward systems within reinforcement learning (RL), where model outputs are evaluated against ground truth references. However, existing reward benchmarks focus on preference comparisons between responses rather than evaluating verification against ground truth references, leaving a critical gap in our ability to evaluate verification systems used in reasoning model training. In this paper, we introduce VerifyBench and its challenging variant VerifyBench-Hard, two benchmarks specifically designed to assess reference-based reward systems. These benchmarks are constructed through meticulous data collection and curation, followed by careful human annotation to ensure high quality. Our comprehensive evaluation reveals that while larger model-based verifiers show promise on standard cases, all current systems demonstrate substantial room for improvement on challenging instances. Through systematic analysis of performance patterns across reasoning tasks and error categories, we provide insights for advancing reference-based reward systems. These benchmarks establish a standardized framework for improving verification accuracy, ultimately enhancing reasoning capabilities in models trained via RL. Project Page VerifyBench Benchmark ZJU-REAL/VerifyBench Code ZJU-REAL/VerifyBench 1 Introduction In recent years, large language models (LLMs) have exhibited remarkable capabilities, significantly assisting humans across diverse practical domains ( DeepSeek-AI et al. 2025b ; Grattafiori et al. 2024 ; Yang et al. 2025 ) . Reinforcement learning from human feedback (RLHF) has been crucial to this progress, with reward models playing a central role by evaluating and scoring model-generated responses to guide training. This approach has led to the development of numerous benchmarks ( Lambert et al. 2024 ; Liu et al. 2024 ; Zhou et al. 2025 ) for systematic reward model evaluation, focusing primarily on pairwise preference judgments between competing responses. Figure 1: The core distinction between VerifyBench and existing reward benchmarks ( Lambert et al. 2024 ; Liu et al. 2024 ) is illustrated as follows. Upper panel: Existing reward benchmarks assess the accuracy of a reward system by comparing the ranking of two completions for the same question. Lower panel: In contrast, our proposed VerifyBench evaluates the accuracy of a reward system by determining the correctness of a single completion using a reference answer. The emergence of specialized large reasoning models (LRMs) ( DeepSeek-AI et al. 2025a ; Qwen Team 2024 ; Kimi Team et al. 2025 ) such as OpenAI’s o1 ( OpenAI 2024 ) and DeepSeek-R1 ( DeepSeek-AI et al. 2025a ) has fundamentally changed this landscape. These models achieve unprecedented performance on reasoning tasks through specialized reinforcement learning techniques that differ from standard RLHF approaches. A key distinction in training methodologies for LRMs is their reliance on reference-based reward systems, where rewards are assigned based on alignment between model-generated responses and authoritative reference answers. This approach has been implemented variously across leading models, with Deepseek-R1 ( DeepSeek-AI et al. 2025a ) employs a rule-based reward to prevent reward hacking, whereas models like Seed1.5-Thinking ( Seed et al. 2025 ) adopt model-based reward systems to generate more precise and robust signals. Despite the widespread adoption of reference-based reward systems in training state-of-the-art reasoning models, a significant gap exists in our ability to evaluate these systems systematically. Current benchmarks focus almost exclusively on preference-based evaluation, assessing rewards on their ability to rank competing responses correctly. This approach fails to capture the requirements of reference-based verification, where responses must be judged against objective ground truths rather than relative preferences. The absence of dedicated benchmarks for reference-based reward systems has limited researchers’ ability to assess, compare, and improve their verification methodologies effectively, potentially impeding progress in reasoning model development. To address this critical gap, we introduce VerifyBench, a benchmark specifically designed to evaluate the accuracy of reference-based reward systems. VerifyBench differs fundamentally from existing reward benchmarks by focusing on absolute correctness judgments rather than relative preference assessments. While traditional benchmarks ask reward models to determine which of two responses is better, VerifyBench challenges systems to verify whether a single response correctly aligns with a reference answer, more accurately reflecting the actual use case in reasoning model training. In this paper, we present VerifyBench, a benchmark specifically designed to evaluate the accuracy of reference-based reward systems. To create VerifyBench, we curated a diverse collection of instructions paired with reference answers sourced from existing open datasets. Responses to these instructions were generated by multiple open-source and proprietary LLMs. The correctness of each response was assessed using both automated model judgments and human evaluations. Each instance in VerifyBench was verified by at least two human annotators to ensure label consistency and reliability, thereby producing a high-quality benchmark for the evaluation of reward systems. Recognizing the need to differentiate between various verification techniques and to push the boundaries of current capabilities, we further developed VerifyBench-Hard, a more challenging variant of our benchmark. This dataset focuses on contentious cases where leading models produce highly conflicting judgments, providing a more stringent test for reward system accuracy. VerifyBench-Hard samples were carefully selected based on disagreement patterns among high-performing models, then subjected to thorough human annotation to ensure label quality. Our contributions are summarized as follows: 1. To better reflect realistic reinforcement learning (RL) scenarios for reasoning models, we construct VerifyBench, a benchmark derived from existing models and datasets, to provide an objective evaluation of the accuracy of reference-based reward systems. 2. We further develop VerifyBench-Hard, a more challenging benchmark curated from cases exhibiting high disagreement among multiple models. This dataset contains a larger proportion of difficult-to-verify samples, highlighting substantial potential for improvement in current models. 3. We conduct a comprehensive empirical analysis of model performance on both VerifyBench and VerifyBench-Hard, offering actionable insights to advance the accuracy of reference-based reward systems and enhance RL training in reasoning tasks. 2 Preliminaries Reference-free Reward Models In reinforcement learning (RL) for large language models (LLMs), the reward model plays a crucial role by approximating real-world reward signals associated with model-generated outputs. A typical reward model takes as input a user’s query q along with the corresponding LLM-generated response r , and produces a reward signal, formally defined as: r=R_{\varphi}(q,r) (1) where q represents the user’s query, r denotes the response generated by the LLM, and \varphi encapsulates either the learned parameters of the reward model or the heuristic criteria used to evaluate the quality of the response given q and r . Evaluation of Reference-free Reward Models Generally, reward models produce scalar outputs whose scales can vary significantly across different implementations, complicating direct numerical comparisons. Consequently, current benchmarks evaluate reward models using a pairwise comparative approach. Formally, given a dataset D comprising tuples (q,r_{w},r_{l}) , where q represents a user’s query, and r_{w} and r_{l} denote two candidate responses with r_{w} considered superior to r_{l} , the accuracy of a reward model is quantified as the proportion of instances in which the model correctly assigns a higher score to r_{w} than to r_{l} . Mathematically, this accuracy metric is defined as: \text{Acc}=\frac{1}{|D|}\sum_{(q,r_{w},r_{l})\in D}\mathbb{I}\left[R_{\varphi}(q,r_{w})>R_{\varphi}(q,r_{l})\right] (2) where \mathbb{I}(\cdot) is the indicator function, and R_{\varphi} denotes the