RewardBench 2: Advancing Reward Model Evaluation
Saumya Malik Valentina Pyatkin Sander Land Jacob Morrison, Noah A. Smith Hannaneh Hajishirzi Nathan Lambert · 2025
Read RewardBench 2: Advancing Reward Model Evaluation by Saumya Malik Valentina Pyatkin Sander Land Jacob Morrison, Noah A. Smith Hannaneh Hajishirzi Nathan Lambert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01937.txt.
Introduction
Saumya Malik and co-authors released in 2025 a harder yardstick for reward models, the scorers that grade language model answers in training. RewardBench 2 uses fresh human prompts rather than recycled test items, across many skill areas. Models score about twenty points lower on it than on the first version, which shows its bite. Yet scores on it track downstream training gains closely.
The team details how the set was built and how current models fare. They also tie bench scores to two uses: picking the best of many sampled answers, and classic preference training. The aim is an accuracy test the field can trust when tuning reward models.
Catalog record
| CREATOR | Saumya Malik Valentina Pyatkin Sander Land Jacob Morrison, Noah A. Smith Hannaneh Hajishirzi Nathan Lambert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01937
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RewardBench 2: Advancing Reward Model Evaluation Saumya Malik Valentina Pyatkin Sander Land Jacob Morrison, Noah A. Smith Hannaneh Hajishirzi Nathan Lambert Abstract Reward models are used throughout the post-training of language models to capture nuanced signals from preference data and provide a training target for optimization across instruction following, reasoning, safety, and more domains. The community has begun establishing best practices for evaluating reward models, from the development of benchmarks that test capabilities in specific skill areas to others that test agreement with human preferences. At the same time, progress in evaluation has not been mirrored by the effectiveness of reward models in downstream tasks – simpler direct alignment algorithms are reported to work better in many cases. This paper introduces RewardBench 2, a new multi-skill reward modeling benchmark designed to bring new, challenging data for accuracy-based reward model evaluation – models score about 20 points on average lower on RewardBench 2 compared to the first RewardBench – while being highly correlated with downstream performance. Compared to most other benchmarks, RewardBench 2 sources new human prompts instead of existing prompts from downstream evaluations, facilitating more rigorous evaluation practices. In this paper, we describe our benchmark construction process and report how existing models perform on it, while quantifying how performance on the benchmark correlates with downstream use of the models in both inference-time scaling algorithms, like best-of-N sampling, and RLHF training algorithms like proximal policy optimization. 1 Introduction Reward Models (RMs) are often designed to model human preferences to improve language model training ouyang2022training ; bai2022training ; touvron2023llama ; dubey2024llama . Generally, a reward model is trained to output a scalar value proportional to (some aspects of) the quality of the input text, learned from preference data. RMs have been used extensively for RLHF training nakano2021webgpt ; glaese2022improving , but also are used for online direct alignment algorithms singhal2024d2po , data filtering albalak2024survey ; dubey2024llama , and inference-time scaling faria2025sample ; chow2024inference . Despite extensive use, the ecosystem of directly evaluating reward models is still nascent and developing alongside the roles RMs play. Users developing RMs for their application must decide which benchmark(s) to use. This is a multi-dimensional decision process, as evaluations vary in how they measure performance (e.g., accuracy vs. correlation with LM-as-a-judge) and the domains they focus on (e.g., multi-skill vs. chat-only). The first reward model evaluations such as RewardBench lambert2024rewardbench and RM-Bench liu2024rm focused on simple classification tasks to measure performance of existing reward models across common domains like style and safety. Additional evaluations included analysis of downstream scores when the RM is used within inference-time methods such as best-of-N (BoN) sampling zhou2024rmb and also training with RLHF frick2024evaluate . We present RewardBench 2, a benchmark built on classification tasks that measures and improves correlations relative to earlier approaches of RM evaluations in two scenarios: inference-time compute and downstream training (highlighted in Figure 1 ). Our benchmark maintains strengths of multiple existing benchmarks, such as using unseen human prompts or switching from the common practice of accuracy over a chosen and rejected response to one chosen and three rejected responses to reduce the distance between strong reward models and the random baseline, as summarized in Table 1 . The benchmark covers six domains: three new datasets to improve evaluation in domains covered by existing RM benchmarks – focus, math, and safety – along with three new challenging domains: factuality, precise instruction following, and ties (a new type of domain where we test a RM’s ability to be well-calibrated between equivalently valid answers, like “red” and “green” in response to “Name a color of the rainbow”). In total we evaluate over 100 reward models, a mix of leading existing models and new models we trained to better understand the relationship between RM training and evaluation, in order to allow more reliable use of RMs across a variety of skills often targeted in post-training in order to allow more reliable use of reward models. The benchmark was created with a majority of previously unused human prompts from the WildChat pipeline ( zhao2024wildchat ) with extensive manual, programmatic, and LM-based filtering techniques. To validate the benchmark, we run extensive experiments to show how RM benchmarks can be used in effective RLHF training workflows or correlated hillclimbing targets for inference-time compute techniques. Figure 1 : RewardBench 2 is composed of high-quality, unseen human prompts designed for a best-of-4 reward model evaluation format with completions generated from a variety of leading AI models. We extend RM evaluation of pairwise “chosen” and “rejected” completions to include additional rejected samples as distractions. RewardBench 2 has 6 domains which expand upon challenging domains in existing RM evaluations and adds a new domain, Ties, to test how RMs handle questions with multiple correct answers. The new data and setup enables more accurate correlation of benchmark scores with downstream performance via RL finetuning or best-of-N sampling. Our contributions and findings are as follows: 1 1 1 Code for RewardBench 2 is available here: https://github.com/allenai/reward-bench . The benchmark dataset is available here: https://huggingface.co/datasets/allenai/reward-bench-2 . 1. RewardBench 2 provides a challenging evaluation of reward models across many domains on majority unseen prompts with leading models on the first version of RewardBench scoring 20 or more points lower on RewardBench 2. This includes challenging subsets such as Precise Instruction Following and Math where leading models are below 40% and 70% accuracy, respectively with data details discussed in Section 3 . 2. Controlled experiments where we train reward models and analyze their performance on the benchmark, gaining actionable insights for reward model training . In particular, we find that different post-trained base models, even within the same lineage and model family, offer different capabilities to reward models and that, contrary to the accepted best practice, training for more than one epoch can be beneficial. We discuss these findings in Section 4 . 3. An exploration of the benefits and limits of using a reward model evaluation to inform downstream use cases of inference-time scaling algorithms and RLHF training. In Section 5 our benchmark achieves strong downstream correlation with inference time scaling algorithms like best-of-N sampling and provides a helpful signal for PPO training. 4. Our analysis shows how the best reward model for RLHF is dependent on one’s training setup. For RLHF, the reward model should be based on a model of the same lineage as the policy model or else downstream performance can degrade significantly, so simply taking the highest scoring reward model on a benchmark will not ensure a good post RLHF model. 2 Background Reward Models Reward models are trained on preference data, consisting of prompts x and completions y_{i} , where each completion has been ranked by humans or automated metrics like ground truth signals and language model judgments lambert2025reinforcement . The canonical formulation, which we use in this work, is to create preference pairs , where for each prompt two completions are compared, and the better prompt is “chosen”, and the other is “rejected.” With that data, a reward model r^{*} is trained to output a scalar value to predict the probability p^{*} of a prompt and completion falling in the chosen category, following a Bradley-Terry model of human preferences ( BradleyTerry ) : p^{*}(y_{1}\succ y_{x}\