JudgmentBench : Comparing Rubric and Preference Evaluation for Quality Assessment
Russell Yang 1 Ruishi Chen 1 Pierce Kelaita 1 Riya Ranjan 1 Sibo Ma 1 Charles Dickens 2 Matthew Guillod 3 Megan Ma 1 Julian Nyarko 1 1 Stanford University 2 Snorkel AI 3 Harvey [email protected] Dataset: huggingface.co/datasets/judgmentbench/JudgmentBench Code: github.com/liftlab-SLS/JudgmentBench · 2026
Read JudgmentBench : Comparing Rubric and Preference Evaluation for Quality Assessment by Russell Yang 1 Ruishi Chen 1 Pierce Kelaita 1 Riya Ranjan 1 Sibo Ma 1 Charles Dickens 2 Matthew Guillod 3 Megan Ma 1 Julian Nyarko 1 1 Stanford University 2 Snorkel AI 3 Harvey [email protected] Dataset: huggingface.co/datasets/judgmentbench/JudgmentBench Code: github.com/liftlab-SLS/JudgmentBench (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25240.txt.
Introduction
Russell Yang and colleagues build a test that pits two ways of judging answers against each other on real legal work. JudgmentBench gathers 30 tasks with scores from practicing lawyers, both as fixed-criteria grades and as head-to-head preference picks. When the same experts supply both kinds of judgment, side-by-side choices recover the intended quality ranking. They do far better than rubric grades. Preference picks also take less than half the grading time, a gap that holds for human graders and automated ones alike.
Catalog record
| CREATOR | Russell Yang 1 Ruishi Chen 1 Pierce Kelaita 1 Riya Ranjan 1 Sibo Ma 1 Charles Dickens 2 Matthew Guillod 3 Megan Ma 1 Julian Nyarko 1 1 Stanford University 2 Snorkel AI 3 Harvey [email protected] Dataset: huggingface.co/datasets/judgmentbench/JudgmentBench Code: github.com/liftlab-SLS/JudgmentBench |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69730 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25240
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
JudgmentBench : Comparing Rubric and Preference Evaluation for Quality Assessment Russell Yang 1 Ruishi Chen 1 Pierce Kelaita 1 Riya Ranjan 1 Sibo Ma 1 Charles Dickens 2 Matthew Guillod 3 Megan Ma 1 Julian Nyarko 1 1 Stanford University 2 Snorkel AI 3 Harvey [email protected] Dataset: huggingface.co/datasets/judgmentbench/JudgmentBench Code: github.com/liftlab-SLS/JudgmentBench Abstract Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench , a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys–including at major U.S. law firms–with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same experts on the same items. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics under both a per-task rank-correlation metric (mean Spearman’s rank correlation of 0.908 vs. 0.150, D ^ ρ = 0.758 \widehat{D}_{\rho}=0.758 [0.494, 1.021]) and a per-judgment pairwise win-rate metric (0.669 vs. 0.542, D ^ W = 0.127 \widehat{D}_{W}=0.127 [0.067, 0.186]), while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth. 1 Introduction Large language models (LLMs) are increasingly deployed in high-stakes, high-expertise domains like medicine ( Singhal et al. 2023 ; Pierson et al. 2025 ; Omiye et al. 2024 ) , law ( Guha et al. 2023 ; Fei et al. 2024 ) , and finance ( Wu et al. 2023 ; Yang et al. 2023 ; Xie et al. 2023 ) . Many of the underlying scenarios are particularly judgment rich, rendering the measurement of output quality of these models notoriously difficult. Practical approaches vary widely across investigators and contexts, with two methodologies dominating current evaluation efforts: Under rubric-based scoring, outputs are graded against predefined criteria ( Arora et al. 2025 ; Shi et al. 2026 ; Sharma et al. 2026 ; Brookhart 2018 ; Jonsson and Svingby 2007 ) . In contrast, under comparative judgment (CJ), annotators compare two outputs and identify the one they prefer ( Thurstone 1927 ; Pollitt 2012 ; Lesterhuis et al. 2017 ; Verhavert et al. 2019 ) . In the benchmarking literature, the choice between the two methodologies often appears arbitrary and is rarely justified, raising questions as to whether and when one approach might yield more reliable assessments over the other. We introduce a new type of benchmark dataset designed to allow for direct comparisons of rubric-based scoring and comparative judgment as assessment methodologies. Our study is grounded in the legal domain, capturing realistic work processes in large-scale commercial legal practice. This setting is often characterized by high levels of complexity in which attorneys accumulate and rely on experience acquired over many years of practice. Defining “quality” in such a setting is particularly challenging as legal work products are frequently multidimensional, context-dependent and resist straightforward objective evaluation criteria ( Ma et al. 2023 ; Scott 2009 ) . For example, outcomes alone are a poor proxy for quality in summary judgment briefing. Strong briefs may fail due to adverse facts or judicial preferences, while weaker ones may prevail when the law clearly favors one side ( Tippett et al. 2022 ) . In contexts like these, the choice of assessment methodology is itself part of the problem of defining legal quality. The significance of comparing rubric-based scoring with comparative judgment extends beyond benchmark design. As LLMs are applied to economically consequential knowledge work, expert judgment must be represented in forms that can support systematic methods (e.g., evaluation, quality assurance, model post-training) at scale. Rubric-based scoring and comparative judgment represent two different approaches to this conversion. Rubrics are costly to create because they require experts to specify the dimensions of quality in advance, but once created, it is often assumed they are easy to automate and scale across systems ( Starace et al. 2025 ; Liu et al. 2025 ) . Comparative judgments, by contrast, may better preserve tacit and holistic dimensions of expertise, including nuance, creativity, strategic judgment, and persuasive force, especially where quality is difficult to enumerate in advance ( Jonsson and Svingby 2007 ; Pollitt 2012 ) . Yet because pairwise preference judgments are typically aggregated into rankings or latent utility estimates, the corresponding evaluations are harder to decompose into reusable criteria or audit after aggregation. This tradeoff matters not only for how AI systems are evaluated, but also for how expert judgment is incorporated into downstream systems such as model post-training. Against this backdrop, our contributions are threefold: 1. A benchmark dataset of real-world tasks. We release a set of high-complexity tasks drawn from Harvey’s BigLaw Bench, selected to be representative of economically valuable legal work rather than stylized exam-style problems. 2. Paired rubric and preference annotations from legal experts. Each task is annotated by experienced, practicing lawyers along both dimensions — atomistic, expert-developed rubric criteria and pairwise preference judgments — representing approximately $242,000 in equivalent billable attorney time. 1 1 1 We base this estimate on roughly 242 hours of attorney time, billed at an average of $1,000 per hour. To our knowledge, this is the first dataset in any high-expertise domain in which both supervision signals are elicited from the same experts on the same items. This pairing is the precondition for any clean comparison between the two approaches. 3. A direct empirical comparison of rubric-based scoring and comparative judgment in law. We find that comparative judgments are consistently better than rubrics for recovering the constructed quality ordering while requiring less than half the annotation time. The result holds for both human annotators and LLM autograders, and has important implications for how quality should be measured and how reward signals should be constructed in high-stakes professional domains. 2 Background and Related Work 2.1 Conceptual Foundations Benchmarks are essential infrastructure for evaluating the quality of LLM outputs, providing standardized conditions under which models can be compared, progress tracked, and failure modes surfaced ( Liang et al. 2023 ; Chang et al. 2024 ; Qian et al. 2026 ; Reuel et al. 2024 ) . The reliability of any benchmark, however, depends not only on the tasks it includes but on the methodology it uses to elicit and aggregate human or model judgments. These methodological choices long predate the era of LLM evaluation and have been studied extensively in the educational ( Jonsson and Svingby 2007 ; Brookhart 2018 ; Pollitt 2012 ; Verhavert et al. 2019 ) and psychological ( Thurstone 1927 ; Maydeu-Olivares and Böckenholt 2005 ; Stellmack et al. 2009 ) measurement literature. Two strategies have historically dominated: rubrics, in which a response is scored against a set of predefined criteria, and comparative judgments, in which pairwise preferences are aggregated into an ordering. Rubrics rest on the premise that quality can be made explicit by