Evaluating Judges as Evaluators: The Jetts Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, Shafiq Joty · 2025

Read Evaluating Judges as Evaluators: The Jetts Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators by Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, Shafiq Joty (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15253.txt.

Introduction

Yilun Zhou and four co-authors present a 2025 benchmark study of language models used as judges. At test time, a generator model can spend extra compute to improve answers, often guided by a separate scoring model. Judge models instead write critiques in plain words, and their worth in that guide role was unknown.

The benchmark, called JETTS, tests judges in math, code, and instruction tasks. Three jobs are covered: ranking finished replies, guiding step-by-step search, and refining replies from critiques. Ten judge models faced eight generator models. Judges matched scoring models at ranking, but lost to process scorers in search. Their written critiques did not yet guide generators to better replies.

Catalog record

CREATORYilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, Shafiq Joty
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE139 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Judges as Evaluators: The Jetts Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, Shafiq Joty Abstract Scaling test-time computation, or affording a generator large language model (LLM) extra compute during inference, typically employs the help of external non-generative evaluators (i.e., reward models). Concurrently, LLM-judges, models trained to generate evaluations and critiques (explanations) in natural language, are becoming increasingly popular in automatic evaluation. Despite judge empirical successes, their effectiveness as evaluators in test-time scaling settings is largely unknown. In this paper, we introduce the Judge Evaluation for Test-Time Scaling ( Jetts ) benchmark, which evaluates judge performance in three domains (math reasoning, code generation, and instruction following) under three task settings: response reranking, step-level beam search, and critique-based response refinement. We evaluate 10 different judge models (7B-70B parameters) for 8 different base generator models (6.7B-72B parameters). Our benchmark shows that while judges are competitive with outcome reward models in reranking, they are consistently worse than process reward models in beam search procedures. Furthermore, though unique to LLM-judges, their natural language critiques are currently ineffective in guiding the generator towards better responses. † † affiliationnotice: Equal contribution https://github.com/SalesforceAIResearch/jetts-benchmark 1 Introduction Judge Reranking \uparrow Beam Search \uparrow Refinement \uparrow Prometheus-2 7B -0.107 -0.100 0.976 SFR-Judge 8B 0.035 0.001 0.941 Skywork-Critic 8B 0.039 0.046 - OffsetBias 8B -0.020 0.015 - Themis 8B -0.149 -0.021 0.996 SFR-Judge 12B 0.013 0.047 0.943 Prometheus-2 8x7B -0.079 -0.083 0.966 SFR-Judge 70B 0.171 0.138 0.951 Skywork-Critic 70B 0.177 0.132 - Self-Taught Eval. 70B 0.095 0.072 - Best RM 0.113 - - Best PRM - 0.195 - Random -0.193 -0.139 - Baseline Value 0.000 0.000 1.000 Figure 1: Top: a graphical description of the three test-time compute tasks in Jetts used to evaluate LLM-judges. Bottom: the leaderboard of judges on the three benchmark tasks, along with outcome/process reward models and random choice. “-” means that the model cannot be used for the task. Achieving a value less than the baseline (grayed out) means that the judge is worse than some less compute-intensive approach (e.g., greedy decoding, see Sec. 3.5 for details). For the last several years, the rapid increase in various capabilities of large language models (LLMs) is mostly attributed to scaling – the corresponding increase in model and training data sizes ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ) . Nonetheless, this effort has gradually saturated due to lack of additional data and prohibitive cost of model training and serving. Recently, test-time scaling is being considered as an alternative solution, in which more compute is dedicated at the test time in the hope that better model responses are generated ( Snell et al., 2024 ; Jaech et al., 2024 ) . A common component in test-time compute is the evaluator model, which gives a quality signal to various generated responses. Scalar reward models (RMs) are predominantly used to rerank either complete responses in best-of-N selection or partial responses in step-wise generation (e.g., beam search). By contrast, generative LLM judges are often favored in model evaluation, e.g., AlpacaEval ( Dubois et al., 2024b ) and MT-Bench ( Zheng et al., 2023 ) , since they can adapt to custom evaluation criteria and can (often) produce natural language critiques (or explanations) of the response. Despite LLM-judges achieving competitive performance on reward model benchmarks (e.g., RewardBench ( Lambert et al., 2024 ) ) and PPE ( Frick et al., 2024 ) ), they are less commonly used in test-time scaling scenarios compared to scalar RMs ( Ji et al., 2025 ; Ke et al., 2025 ) . This gap is perhaps surprising given the potential benefits of LLM-judges. Namely, judge models are often trained to engage in chain-of-thought reasoning before giving a judgment, which could not only benefit for reasoning-intensive domains such as math and code, but also serve as effective critiques for the model to refine its generation, such as in Reflexion ( Shinn et al., 2024 ) or SCORE ( Zhang et al., 2024b ) . Furthermore, LLM-judges could seamlessly integrate tool use. Last, LLM-judges are inherently “instructable”, capable of giving different types of judgment like binary yes/no answers, numerical scores, fine-grained feedback, etc., simply with different prompts. In this paper, we propose the first systematic benchmark of LLM-judges for model’s test-time scaling. This benchmark consists of three tasks (Fig. 1 ): (1) response reranking, where the judge picks the best response out of several ones, (2) step-level beam search, where the judge guides the model to generate the response step-by-step, and (3) critique-based refinement, where the judge offers natural language critique to the response for the model to refine. We evaluate 10 different judge models, ranging in size from 7B to 70B parameters on these three test-time scaling tasks. We cover 3 different domains (math reasoning, code generation, and instruction following) and generate responses from up to 8 unique LLMs, ranging in size from 6.7B to 72B parameters. The Jetts setup enables us to analyze the effects of judge-guided test-time scaling: Can a “weak” (e.g., 8B) judge actually help a “strong” (e.g., 72B) generator? How beneficial are strong judges to weak generators? In what domains are the current crop of judge models best suited for test-time scaling? How useful are judge critiques in practice? We find that weak judges can help strong generators in easier tasks, such as instruction following, but not in reasoning-intensive tasks like coding or math. Larger judges bring the most benefit for math and instruction following tasks, but no evaluated judges are able to reliably improve generator performance for coding. Lastly, while natural language critiques are touted as a defining advantage of judges over RMs, we find that such critiques have significant room for improvement in terms of utility. We present these results with many additional analyses in Sec. 4 . Figure 2: Judge performance on RewardBench vs. Jetts , where marker size signifies judge model size (7B-70B). Small judges perform comparably to large judges on RewardBench, but lag large judges on Jetts , which simulates test-time scaling settings. Comparison with RewardBench. In Fig. 2 , we present a preview of our results by comparing performance on JETTS Best-of-N Reranking and Beam Search tasks against RewardBench ( Lambert et al., 2024 ) , a popular RM and judge model benchmark. While judge performance between the two benchmarks is generally correlated, Jetts reveals a difference in the “fundamental judging ability” between judges of different scales. For example, if deciding between Skywork-Critic-8B and 70B, the practical choice based on RewardBench is to use the 8B model, trading off 4% in accuracy for an order of magnitude fewer parameters. Jetts , which simulates test-time scaling scenarios, reveals that this choice is suboptimal for reranking: The 8B judge yields substantively lower improvements over the greedy response than the 70B judge. The benchmark construction approach likely contributes to the difficulty gap: RewardBench forms pairs of responses from different generators, meaning judges can arrive at the right outcome via stylistic factors, like formatting ( Tan et al., 2024 ) . Jetts , however, requires judges to compare responses sampled from the same generator, limiting stylistic factors that lead to correct outcomes. 2 Background and Related Work The concept of using

Read the complete text