Bias and Uncertainty in LLM-as-a-Judge Estimation

James Fiedler · 2026

Read Bias and Uncertainty in LLM-as-a-Judge Estimation by James Fiedler (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06939.txt.

Introduction

Fiedler examines a quiet flaw in a popular way of grading language models, where one model is used as a judge of other models. Raw judge scores are systematically biased. Researchers correct them with calibration data, but corrections are only as good as the judge behind them. Sharing one calibration across compared models looks efficient, yet it can badly distort the comparison, even flipping which model appears better while looking highly confident.

Through analysis and simulations that vary judge quality and calibration drift, the author maps when corrected estimates go wrong. A real data MMLU-Pro case study shows an actual sign reversal. He proposes two diagnostics, judge quality J and cross model calibration shift, called delta J, to flag comparisons that are likely unreliable. He closes with reporting guidance so judge based evaluations state their uncertainty instead of presenting a biased score as a clean measurement.

Catalog record

CREATORJames Fiedler
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06939

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bias and Uncertainty in LLM-as-a-Judge Estimation James Fiedler Abstract LLM-as-a-Judge evaluation has become a standard tool for assessing base model performance. However, characterizing performance via the naive estimator, i.e., raw judge outputs, is systematically biased. Recent work has proposed estimators to correct this bias, but their reliability depends critically on judge quality and, for model comparisons, on calibration stability. Sharing calibration across compared models is practically attractive but can introduce severe bias, including cases where the comparison estimate points in the wrong direction with high apparent confidence. We study these failure modes through analytical results, simulations over judge quality ( J J ) and cross-model calibration instability ( Δ ​ J \Delta J ), and a real-data MMLU-Pro case study with sign reversal. We propose J J and Δ ​ J \Delta J as diagnostics for when corrected estimates, especially shared-calibration comparisons, are likely unreliable, and provide reporting guidance for LaaJ evaluation. 1 Introduction LLM-as-a-Judge (LaaJ) evaluation has become a standard tool for assessing base model performance across a growing range of tasks and benchmarks ( Zheng et al. 2023 ; Liu et al. 2023 ; Dubois et al. 2024 ) . Raw judge outputs are often treated as direct measurements of model quality; reporting these outputs without correction or confidence intervals has become common practice ( Lee et al. 2026 ) . The core statistical problem is not only bias, but whether corrected judge outputs can be trusted as estimates of true model performance. Bias-corrected estimators ( Lee et al. 2026 ; Chen et al. 2026 ) can substantially improve estimates, but they are not automatically trustworthy. Their reliability depends on judge quality, summarized by Youden’s J , and on whether the calibration assumptions required to justify the correction actually hold. The central issue here is not estimation efficiency, but whether the resulting estimate and uncertainty statement correspond to the intended quantity. We therefore treat LaaJ estimation as an evaluation method that should itself be stress-tested before it is used to support scientific claims. Lee et al. 2026 show that the Rogan-Gladen estimator (RG) can remain valid under distribution shift when judge predictions depend only on latent correctness rather than, for example, properties of the base model being evaluated. This invites a practical strategy of reusing calibration estimates across multiple evaluation contexts, to save on labeling time and cost. However, such reuse is only justified if judge calibration is effectively invariant across those contexts. When it is not, estimation can be severely distorted. For model comparison, the distortion can be large enough to reverse the sign of the estimate, producing an apparently confident conclusion that points in the wrong direction. This paper studies LaaJ estimation for base-model assessment, covering both single-model accuracy estimation and model comparison, with particular attention to comparison because it is more fragile. Our analytical and simulation results focus on the case where the judge outputs a binary label; it is the most analytically transparent setting and is one of, if not the, most common in practice. However, the core lessons apply more broadly, including to ordinal and continuous judge outputs. We study these failure modes through three lines of evidence. Analytical results characterize the bias structure under shared calibration and the role of J in determining estimator stability, where J=\mathrm{sensitivity}+\mathrm{specificity}-1 is Youden’s index ( Peirce 1884 ; Youden 1950 ) . Simulations demonstrate the impact that judge quality ( J ) and cross-model calibration instability ( \Delta J ) can have on bias and interval coverage. We follow up with a real-data case study using two subjects from the MMLU-Pro benchmark ( Wang et al. 2024c ) that corroborates and expands on the simulation. In one subject estimators perform relatively well. The other is more severe and reveals additional failures, including a sign-reversal failure, where a shared-calibration comparison yields a confidently negative estimate for a truly positive accuracy difference. Our contributions are: 1. We characterize the fragility of bias-corrected LaaJ estimators, showing that estimates for both single-model quality and model comparison can degrade sharply as judge quality ( J ) falls, and that the severity and form of failure differ across estimators. 2. We analyze shared calibration as a risk factor in both single-model and comparison estimation, showing that it induces 1/J sensitivity to calibration mismatch and, for RG, can yield severely amplified systematic bias when J is small. 3. We propose J and \Delta J as prerequisite diagnostics that should be checked and reported before trusting corrected LaaJ estimates, with increasing caution with decreasing J or increasing \Delta J . 4. We demonstrate estimator failure modes through simulation and a real-data MMLU-Pro case study, and provide practical reporting guidance for LaaJ evaluation. 2 Related work LLM-as-a-Judge (LaaJ) evaluation has been widely adopted since MT-Bench ( Zheng et al. 2023 ) and G-Eval ( Liu et al. 2023 ) helped establish the use of LLMs to grade model responses at scale. Subsequent work has extended LaaJ to preference-style evaluation ( Dubois et al. 2024 ) and created structured judge benchmarks ( Tan et al. 2025 ) . Across this literature, naive judge scores are routinely treated as estimates of base-model quality without calibration correction or confidence intervals ( Zheng et al. 2023 ; Tan et al. 2025 ; Lee et al. 2026 ) . Miller 2024 is a strong precedent for statistical uncertainty reporting in LLM evaluation more broadly. Our focus is on calibration challenges specific to LaaJ that make naive uncertainty statements unreliable even after accounting for sampling variation. Lee et al. 2026 showed that naive LaaJ estimates are biased and proposed applying the Rogan-Gladen estimator ( Rogan and Gladen 1978 ) to correct for judge sensitivity and specificity. They establish that the RG estimator remains valid under test-set distribution shift when the judge’s predictions depend only on latent correctness, and validate on Chatbot Arena win-rate data ( Chiang et al. 2024 ) . Chen et al. 2026 subsequently compared RG, PPI ++ ( Angelopoulos et al. 2024 ) , and an EIF-based estimator, showing that PPI ++ and the EIF estimator are more efficient than RG in the binary case. Our focus differs: rather than efficiency, we study when corrected estimates break down, especially in model comparison under shared calibration and cross-model instability. Neither paper studies shared calibration, proposes diagnostics for estimator trustworthiness, or foregrounds the structural distinction between judge-centric shared calibration and model-specific calibration that shapes our analysis. The epidemiological literature has long recognized that the Rogan-Gladen estimator should not be applied indiscriminately. Greiner 2003 showed via second-order Taylor-series analysis, confirmed by simulation, that the corrected estimate can have higher mean squared error than the naive estimate. Lang and Reiczigel 2014 derived adjusted confidence intervals accounting for uncertainty in sensitivity and specificity, establishing that the variance of the RG estimator scales as 1/J^{2} . This instability is intrinsic to the RG formula but not to the estimation problem itself: PPI ++ degrades more gracefully as judge quality falls. Both papers address single-population estimation. We extend to model comparison, where shared calibration can introduce systematic bias when judge calibration differs across models, a failure mode we are not aware of prior work characterizing. Within LLM evaluation literature, Chen et al. 2026 find in

Read the complete text