Bridging Human and LLM Judgments: Understanding and Narrowing the Gap

Felipe Maia Polo, Xinhe Wang, Mikhail Yurochkin, Gongjun Xu, Moulinath Banerjee, Yuekai Sun · 2025

Read Bridging Human and LLM Judgments: Understanding and Narrowing the Gap by Felipe Maia Polo, Xinhe Wang, Mikhail Yurochkin, Gongjun Xu, Moulinath Banerjee, Yuekai Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12792.txt.

Introduction

Felipe Maia Polo and five co-authors build a statistical bridge between human and machine judges in a 2025 paper. Language models now grade model outputs at scale, but their marks drift from human marks in set ways. Their frame, Bridge, posits a hidden human taste score behind each prompt and reply pair, then models the machine's drift as a straight shift on measured traits.

One fitting method serves both lone scores and head to head picks, with math promises for its reads. Six model judges faced two big benches. After bridging, machine marks agreed with human marks more closely on accuracy, calibration, and gap size. The shifts also expose where machine taste parts from human taste in plain terms. The authors offer the frame as shared ground for judge audits.

Catalog record

CREATORFelipe Maia Polo, Xinhe Wang, Mikhail Yurochkin, Gongjun Xu, Moulinath Banerjee, Yuekai Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-12792

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Human and LLM Judgments: Understanding and Narrowing the Gap Felipe Maia Polo, Xinhe Wang, Mikhail Yurochkin, Gongjun Xu, Moulinath Banerjee, Yuekai Sun Abstract Large language models are increasingly used as judges (LLM-as-a-judge) to evaluate model outputs at scale, but their assessments often diverge systematically from human judgments. We present Bridge 1 1 1 Please check our GitHub repository: https://github.com/felipemaiapolo/bridge , a unified statistical framework that explicitly bridges human and LLM evaluations under both absolute scoring and pairwise comparison paradigms. Bridge posits a latent human preference score for each prompt-response pair and models LLM deviations as linear transformations of covariates that capture sources of discrepancies. This offers a simple and principled framework for refining LLM ratings and characterizing systematic discrepancies between humans and LLMs. We provide an efficient fitting algorithm with asymptotic guarantees for statistical inference. Using six LLM judges and two benchmarks (BigGen Bench and Chatbot Arena), Bridge achieves higher agreement with human ratings (accuracy, calibration, and KL divergence) and exposes systematic human-LLM gaps. 1 Introduction Accurate and reliable evaluation is fundamental to the advancement and deployment of artificial intelligence (AI) systems, such as Large Language Models (LLMs). Traditional expert-based or automatic methods ( e.g. , ROUGE [ 27 ] or BLEU [ 33 ] ) for open-ended generated text often struggle with either scalability or poor quality. Recently, LLMs have shown significant promise in addressing these challenges through the emergent “LLM-as-a-Judge” (LLMJ) paradigm [ 12 , 22 , 23 ] . Leveraging their extensive knowledge, flexible reasoning, instruction following, and natural language understanding, LLMs can effectively evaluate complex tasks by scoring, ranking, or selecting among diverse outputs. However, ensuring the trustworthiness, robustness, and human-alignment of LLM-based evaluation systems remains a critical challenge that necessitates careful attention. A crucial step towards better judges involves deepening our understanding of LLMJ systems and recognizing their inherent strengths and limitations. Such knowledge allows practitioners to better quantify associated risks and steer the development of more robust evaluation methods. Recent research has extensively explored factors contributing to inaccuracies in LLM judgments and proposed ways to mitigate them. For example, various biases have been well-documented, including preferences for lengthier responses, overly generous scoring tendencies, or biases influenced simply by the presentation order during pairwise comparisons [ 8 , 41 , 48 , 39 ] . In this work, we introduce Bridge , a statistical framework that explicitly connects LLM ratings to human judgments, providing deeper insight into the sources of human-LLM discrepancies and enabling more reliable alignment between the two. Our framework is LLM-agnostic, does not require access to model weights, and can be applied on top of any API. Bridge combines a statistical model with a specialized fitting algorithm via the proposed logit trick , and we demonstrate the asymptotic normality of the resulting estimators. Our model assumes that both human annotators and LLM judges score each prompt-response pair according to a shared latent preference signal, while systematic deviations in LLM scores are captured by a linear transformation of covariates that encode potential human-LLM divergence sources ( e.g. , response length, text sentiment, writer’s creativity). This formulation enables simultaneous, rigorous, interpretable estimation and testing of multiple discrepancies between human and LLM judgments ; something which current approaches cannot accomplish. In addition, it enables lightweight post-hoc corrections: with only a small set of human labels, we can recalibrate LLM scores for improved probabilistic alignment with human assessments. In summary, our contributions are: 1. Proposing Bridge , a statistical framework connecting human and LLM judgments, which combines a statistical model with a specialized fitting algorithm via the proposed logit trick . Our approach (i) allows practitioners to better understand what makes humans and LLMJ different, (ii) enables better probabilistic alignment with human judgments, and (iii) is LLM-agnostic, making it applicable on top of any API. 2. Deriving the asymptotic distribution of our parameter estimators. These asymptotic distributions allow us to construct confidence intervals for our parameters and formally test for different types of human-LLM gaps. Moreover, it allows us to construct confidence intervals for predictive quantities such as the probability of humans making a certain judgment. 3. Validating our framework using six different LLM judges and queries from BigGen Bench and Chatbot Arena. We show that we can better align LLMs to human annotators using a few labeled data points and interpret the systematic differences between the two types of judges. 1.1 Related work Improving the alignment between LLM-based judges and human annotators is often an important factor for their reliable use at scale. Prior work has pursued several complementary strategies: supervised fine-tuning on human-labelled data [ 14 ] ; supplying in-context examples to the judge [ 48 ] ; post-hoc smoothing of raw scores [ 28 , 21 ] ; decomposing complex evaluation tasks into simpler, verifiable criteria [ 38 , 44 ] ; harnessing reference answers or detailed rubrics [ 17 , 47 ] , or chain-of-thought (CoT) strategies to make use of the model’s reasoning abilities [ 17 , 25 ] . A parallel line of research first diagnoses systematic biases in LLMJ and then proposes corrective measures. For instance, Dubois et al. 2024 reveals a strong preference for longer responses and introduces the length-controlled AlpacaEval [ 26 ] benchmark. Park et al. 2024 catalogue six distinct bias types, construct counter-biased datasets, and fine-tune judges on these counter-examples. Additional studies on LLMJ biasing factors document, for example, position bias [ 46 , 48 , 39 , 43 ] , leniency bias [ 41 ] , and sentiment or authority biases [ 46 ] . Few works have also uncovered biases in human ratings themselves [ 5 , 24 ] . On a related but different direction, Buyl et al. 2025 studies how discretion is exercised by humans and LLMs when making choices based on key guiding principles. Despite recent advances, no prior work has systematically examined divergences between human and LLM judgments in a comprehensive way. These discrepancies may carry negative connotations ( e.g. , LLM biases), be value-neutral, or even positive ( e.g. , human biases). Yet such a comparison is essential: it clarifies the strengths and limitations of LLM judges and underscores that corrections should be made relative to human preferences, not in absolute terms, if human alignment is the goal. Otherwise, eliminating a feature valued by both humans and LLMs could inadvertently widen the misalignment. To fill this gap, our work offers a principled way to analyze the systematic differences between human and LLM judges. 2 Problem setup We consider two evaluation scenarios: absolute and relative ratings. In the absolute case, an entity under evaluation ( e.g. , an LLM or a human) is presented with an input prompt I and produces a text output O 1 1 1 While O may take other forms in principle, we restrict our attention to textual outputs. . Given the pair (I,O) , both a human evaluator and an LLM judge assign scalar ratings Y^{h},Y^{l}\in{\mathbb{R}} , where the numerical values reflect absolute assessments, such as whether the response is satisfactory. In the relative case, two entities generate responses O_{A} and O_{B} to the same prompt I , and evaluators provide comparative judgments Y^{h} and Y^{l} that encode one of the options in

Read the complete text