How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation
Wilson Y. Lee · 2026
Read How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation by Wilson Y. Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-09084.txt.
Introduction
Wilson Lee asks how many human picks it takes to grade two chat models fair. Such grades ask folk which reply they like more. Labs use them to claim a new model is a bit more good. But a bit is hard to see. If the gain is spread thin over all ask types, no smart pick of asks can find it fast.
Lee proves that in the thin spread case, a plain share of picks is as good as it gets. No scheme can beat it by much. That sets a floor on grade cost for small gains. A reader who grades models gets a bound to plan by. Small claims need vast pick counts, by law, not by poor design.
Catalog record
| CREATOR | Wilson Y. Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-09084
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation Wilson Y. Lee Abstract Human preference evaluations are widely used to compare generative models, yet it remains unclear how many judgments are required to reliably detect small improvements. We show that when preference signal is diffuse across prompts (i.e., all prompt types are similarly informative), proportional allocation is minimax-optimal: no allocation strategy substantially improves detectability. Empirical analysis of large-scale human preference datasets shows that most comparisons fall into this diffuse regime, exhibiting small preference margins that require far more judgments than typically collected, even in well-sampled comparisons. These limits persist across evaluation protocols and modalities, including chat, image generation, and code generation with execution feedback. In contrast, curated benchmarks that reduce prompt-induced variability systematically induce larger margins and improve detectability through a 1.5 × 1.5\times reduction in prompt-level variance. Our results show that inconclusive or negative human evaluation outcomes frequently reflect underpowered evaluation rather than model equivalence, underscoring the need to account explicitly for effect size, budget, and protocol design. 1 Introduction Human preference studies are widely used to evaluate generative models, particularly for open-ended tasks where automated metrics poorly align with perceived quality ( 11 ; 6 ; 19 ; 13 ) . Pairwise human judgments directly compare models and are central to modern evaluation pipelines, including public leaderboards and curated benchmarks. However, such evaluations are expensive, slow, and often inconclusive, especially as model improvements become incremental. As a result, practitioners routinely face a practical question: “how many human judgments are required to reliably detect a meaningful improvement?” Prior work has noted that human evaluation results are often underpowered or difficult to interpret when effect sizes are small ( 11 ; 8 ; 3 ; 10 ; 17 ) , even under careful uncertainty estimation. Classical power analysis assumes a single homogeneous effect size and i.i.d. samples ( 15 ; 5 ) , which are violated by human preference evaluation where effect sizes vary across prompts ( 19 ) . This motivates a feasibility analysis linking evaluation budget, effect size, and detectability under heterogeneity. Figure 1 : Distribution of empirical absolute preference margins |\delta| for well-sampled Chatbot Arena model pairs ( \geq 200 decisive judgments). A substantial fraction lie in a low-signal regime ( |\delta|\lesssim 0.10 ), where reliable detection requires hundreds of judgments. In this work, we study human preference evaluation through the lens of statistical hypothesis testing under heterogeneous prompts and finite budgets. We show that reliable detection is governed by the total KL divergence accumulated across judgments, and that evaluation protocols determine how this budget is distributed across prompt types. This perspective explains why open-ended evaluations such as Chatbot Arena ( 6 ) often struggle to detect small improvements despite large sample sizes, and clarifies when protocol design can (and cannot) alter feasibility limits. Our contributions. We characterize feasibility in human preference evaluation under prompt-induced heterogeneity, addressing three questions: (1) In what regimes is reliable detection intrinsically difficult? (2) Can adaptive prompt allocation improve detectability, holding the protocol fixed? (3) How does protocol design reshape feasibility limits? Our main theoretical result answers (2) negatively. When preference signal is diffuse across prompts (e.g., open-ended evaluation like Chatbot Arena), proportional allocation is minimax-optimal. In other words, no allocation strategy can rescue underpowered comparisons when the fundamental issue is insufficient total signal. However, when signal is concentrated in a small subset of prompt types, two-stage screening can provide measurable gains. Empirically, we ground this framework using Chatbot Arena and MT-Bench ( 19 ) . Many well-sampled Arena comparisons lie in low-signal regimes requiring far more judgments than typically collected: 17.6% of model pairs with \geq 200 judgments have preference margins |\delta|\leq 0.10 , implying \geq 500 judgments for 90% detection power (Figure 1 ). MT-Bench achieves higher detectability through a 1.5\times reduction in prompt-level variance—providing a mechanistic account of why protocol design, not annotation volume, determines which comparisons are resolvable. 2 Problem Setup and Definitions We consider the problem of comparing two generative models, denoted A and B , using pairwise human preferences. Given a prompt sampled from a fixed distribution and responses generated by both models, a human evaluator is asked to indicate which response they prefer, or whether the responses are tied. Throughout this work, we focus on decisive comparisons and discard ties. Each human judgment is therefore represented as a binary random variable Y=\mathbf{1}\!\left[\text{human prefers model }B\text{ over model }A\right]. (1) where Y=1 indicates a preference for B and Y=0 indicates a preference for A . We define p=\Pr(Y=1), (2) the probability that a randomly sampled evaluator prefers B over A under the given evaluation protocol. Ties and best-case analysis. Human evaluators may also report ties, indicating no clear preference between the two responses. Throughout this work, we discard ties and condition on decisive outcomes in order to obtain a binary preference signal. This choice corresponds to a best-case analysis. Conditioning on decisive judgments maximizes the per-judgment directional information available for distinguishing p\neq 1/2 . Incorporating ties via a multinomial or ordinal preference model can only reduce the effective information per judgment; therefore, it cannot improve detectability. We empirically verify robustness to alternative tie encodings in Appendix D.6 . 2.1 Preference Margin as Effect Size The central quantity governing detectability in pairwise human evaluation is the magnitude of the preference margin. We define the preference margin as \delta=p-\tfrac{1}{2}, (3) which measures the deviation from indifference. Larger values of |\delta| correspond to differences that are easier for human evaluators to perceive, while values of |\delta| close to zero correspond to subtle qualitative differences. From a statistical perspective, detecting an improvement reduces to distinguishing the null hypothesis p=0.5 from the alternative p\neq 0.5 given finite noisy observations. The fundamental quantity governing detectability is the KL divergence between these hypotheses. For Bernoulli preferences near p=0.5 , the per-judgment KL divergence scales as (p-\tfrac{1}{2})^{2} up to constants, so empirical preference margins serve as observable proxies for the KL budget accumulated by an evaluation protocol. 2.2 Near-Tie (Low-Signal) Comparisons We refer to comparisons with small absolute preference margins as near-tie or low-signal comparisons. Formally, for a chosen threshold \tau\in(0,0.5) , we define the near-tie subset as |\hat{p}-\tfrac{1}{2}|<\tau, (4) where \hat{p} is the empirical estimate of p from observed judgments. Rather than treating \tau as a universal constant, we treat it as a sensitivity parameter and report results across multiple thresholds where appropriate. We frame human evaluation as a hypothesis-testing problem, H_{0}:p=0.5\quad\text{vs.}\quad H_{1}:p\neq 0.5, (5) and study detectability as a function of evaluation budget. This framing clarifies that inconclusive human evaluations are not evidence of model equivalence—they may simply reflect insufficient budget relative to the underlying effect size. With n independent judgments, standard binomial tests imply that rejection probability depends on n , the significance level \alpha , and the preference margin |\delta| . Rather than fixing n , we characterize detectability curves mapping budget to detection probability. These curves make explicit the tradeoff between effect size and sample size and clarify the interpretation of inconclusive results: when |\delta| is small, detectability saturates and additional budget yields diminishing returns. 3 Theory: Feasibility Limits and Their Consequences Standard power analysis assumes homogeneous effect size \delta ; the \delta^{-2} sample complexity scaling