A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou · 2026

Read A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth by Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21817.txt.

Introduction

Mingyuan Xu, Xinzi Tan, Jiawei Wu, and Doudou Zhou tackle a growing problem. How to rank large language models when no correct answers exist to check against. A common fix is to let other models act as judges. The catch is that judge models differ a lot in reliability.

Treating all judges as equal can produce biased leaderboards and misleading confidence, the authors note. In fact, more data can make a ranking more confidently wrong when judges are pooled the wrong way. Their 2026 preprint proposes a judge-aware ranking framework that accounts for those differences. Fair ways to weigh judges matter, because public rankings steer which models people buy, fund, and build on.

Catalog record

CREATORMingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21817

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou Abstract Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates—more data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley–Terry–Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings. 1 Introduction Reliable evaluation of large language models (LLMs) has emerged as a central challenge in AI research. As LLMs are increasingly deployed as general-purpose assistants, the ability to accurately rank and compare their capabilities directly impacts both research progress and deployment decisions. Traditional benchmark suites such as MMLU ( Hendrycks et al. 2021 ) and BIG-Bench ( Srivastava et al. 2023 ) provide valuable measurements of specific competencies through tasks with known ground truth. However, these benchmarks often fail to capture user preferences in open-ended, real-world interactions. For instance, chat models fine-tuned with human feedback can be strongly preferred by users while exhibiting minimal differences on traditional benchmarks ( Zheng et al. 2023 ) , highlighting the gap between static evaluation metrics and user satisfaction. To bridge this gap, evaluation protocols based on pairwise comparisons have gained widespread adoption. Human preference evaluation provides a direct signal but is expensive, time-consuming, and difficult to reproduce ( Chiang & Lee 2023 ) . These limitations have motivated the “LLM-as-a-judge” paradigm, wherein LLMs assess outputs produced by other models ( Gilardi et al. 2023 ; Huang et al. 2023 ; Zheng et al. 2023 ; Gu et al. 2024 ) . When carefully prompted, LLM judges can approximate aggregate human preferences on open-ended tasks ( Zheng et al. 2023 ) . Moreover, prompting an LLM to directly compare two responses yields more stable judgments than single-answer grading, making pairwise comparison a natural choice for automated evaluation. As a result, LLM-as-a-judge has become a de facto infrastructure for scalable, reference-free LLM evaluation. A critical but under-examined limitation of this paradigm is that judge reliability varies substantially across models . Empirical studies show that different LLM judges exhibit large variability in their ability to replicate human preferences across tasks ( Bavaresco et al. 2025 ) . Despite this heterogeneity, most existing ranking pipelines implicitly treat all judges as equally reliable and pool their comparisons uniformly. As we show in this work, such aggregation is statistically misspecified : ignoring judge heterogeneity leads to biased rankings and invalid uncertainty quantification. In particular, more data can make the resulting rankings more confidently wrong , as confidence intervals shrink around biased estimates under misspecified models. Classic frameworks such as the Bradley–Terry–Luce (BTL) model ( Bradley & Terry 1952 ; Luce 1959 ) provide a principled foundation for ranking from pairwise comparisons, but assume homogeneous evaluators. Existing approaches for ranking LLMs from automated judgments, such as REM ( Yang et al. 2024 ) and GTR/FTR ( Dhurandhar et al. 2024 ) , either retain this assumption or rely on heuristic weighting schemes that lack a unified statistical interpretation and do not support valid inference. In this paper, we propose a judge-aware ranking framework that extends the BTL model by introducing judge-specific discrimination parameters, jointly estimating latent model quality scores and judge reliability from pairwise comparisons without requiring ground-truth labels. This formulation automatically down-weights unreliable judges through the likelihood and yields well-defined rankings under heterogeneous evaluators. We establish identifiability up to natural normalization constraints and prove consistency and asymptotic normality of the maximum likelihood estimator (MLE), enabling principled uncertainty quantification for LLM rankings, including confidence intervals for score differences and rank comparisons. Empirically, we demonstrate on multiple public benchmarks and a newly collected dataset that judge-aware aggregation improves alignment with human preferences and achieves higher data efficiency than unweighted baselines, particularly in low-budget regimes. Our main contributions are summarized as follows: (1) We identify judge heterogeneity as a fundamental source of bias and invalid uncertainty in LLM-as-a-judge rankings; (2) We propose a judge-aware generalization of the BTL model that jointly estimates model quality and judge reliability without ground-truth labels; (3) We establish identifiability and asymptotic theory for the resulting estimator, enabling valid confidence intervals for LLM rankings; and (4) We demonstrate empirically that judge-aware aggregation improves alignment with human preferences and data efficiency compared to unweighted approaches. The remainder of this paper is organized as follows. Section 2 reviews prior work on statistical ranking models and LLM evaluation. Section 3 formulates the proposed judge-aware model, derives the maximum likelihood estimation algorithm, and establishes asymptotic properties. Section 4 validates the method through numerical simulations. Section 5 reports evaluation results on real-world datasets. Section 6 concludes with discussion and future directions. 2 Related Work 2.1 Statistical Models for Pairwise Ranking Statistical models for pairwise comparisons provide the foundation for ranking problems, with the BTL model ( Bradley & Terry 1952 ; Luce 1959 ) serving as a canonical example. Modern applications have motivated extensions that handle sparse comparison graphs, covariates, and structured heterogeneity. Representative works establish uncertainty quantification under sparse designs ( Gao et al. 2023 ) , develop asymptotic theory for covariate-assisted BTL models ( Fan et al. 2024b ; Yan 2025 ) , and incorporate sparse intrinsic scores or low-rank structure to capture population-level preference heterogeneity ( Fan et al. 2024a ; Fan et al. 2025 ) , with applications to reinforcement learning from human feedback ( Lee et al. 2024 ) . These extensions primarily address heterogeneity in preferences (e.g., different user populations) or comparison structure (e.g., sparse graphs and covariates), while continuing to assume that all observed comparisons are equally reliable. In contrast, we focus on heterogeneity in judge reliability , a distinct and orthogonal source of variation that directly affects ranking validity and uncertainty quantification when comparisons are produced by multiple LLM judges. 2.2 Judge Reliability in Pairwise Ranking Modeling annotator quality has a long history in crowdsourcing and psychometrics. Chen et al. 2013 propose the Crowd-BT model, which captures annotator reliability through an honesty-type parameter corresponding to label flipping, a formulation tailored to settings with random or adversarial workers. Such models are ill-suited to LLM judges, which rarely behave adversarially but instead differ in their ability to resolve fine-grained quality differences. Our model adopts a discriminability-type notion of reliability, in which the parameter \gamma_{k} scales a judge’s sensitivity to quality differences rather than flipping labels. This perspective is closer to Item Response Theory (IRT)-based aggregation models. Otani et al. 2016 introduce judge-specific discrimination parameters for aggregating crowdsourced pairwise evaluations of machine translation systems, but their framework relies on comparisons against fixed references and does not consider fully randomized, reference-free designs. More broadly, existing IRT-inspired

Read the complete text