Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge

Yoshinari Fujinuma · 2025

Read Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge by Yoshinari Fujinuma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18196.txt.

Introduction

Yoshinari Fujinuma, writing alone, shows that language models used as judges bend their scores to whatever range they are given. Asked to grade summaries on a set scale with no reference, judges cluster their marks according to that scale's shape. The same bias appears across models from one family, which makes switching judges a weak fix.

A contrastive decoding trick counters the pull of the range. Agreement with human grades rose on average across ranges in the tests. Evaluation pipelines that lean on model judges get both a named failure mode and a cheap, practical remedy.

Catalog record

CREATORYoshinari Fujinuma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge Thanks: Work done prior and independently of Cantina Labs. Yoshinari Fujinuma Abstract Large Language Models (LLMs) are commonly used as evaluators in various applications, but the reliability of the outcomes remains a challenge. One such challenge is using LLMs-as-judges for direct assessment, i.e., assigning scores from a specified range without any references. We first show that this challenge stems from LLM judge outputs being associated with score range bias, i.e., LLM judge outputs are highly sensitive to pre-defined score ranges, preventing the search for optimal score ranges. We also show that similar biases exist among models from the same family. We then mitigate this bias through contrastive decoding, achieving up to 11.3 % 11.3\% relative improvement on average in Spearman correlation with human judgments across different score ranges. 1 Introduction Large Language Model (LLM) judges have become an integral component of the evaluation ecosystem ( Lin et al. 2022 ; Chiang and Lee 2023 ; Bubeck et al. 2023 ) . In evaluations ranging from direct assessment—where judges evaluate individual outputs by assigning scores— ( Liu et al. 2023 ) to pairwise comparisons—where judges compare two outputs and determine which is superior— ( Zheng et al. 2023 ; Ye et al. 2024 ) , using LLM as a judge is increasingly deployed to provide automatic, scalable, and cost-effective evaluation across diverse tasks. However, the reliability of such evaluations faces significant challenges, particularly when models assess their own outputs ( Zheng et al. 2023 ) or those from the same model family ( Goel et al. 2025 ) . These biases constrain the set of models that can be reliably employed as LLM-as-a-judge for evaluation. But could there be any other biases hidden when using LLMs as judges? Figure 1: Overview of score range bias in 2-4 range and how contrastive decoding mitigates it through canceling out similar bias across models from the same family. In this work, we reveal another bias in LLM judge outputs, namely score range biases , where LLM judge outputs are sensitive to the shift in scores ranges, a phenomenon motivated by prior findings that LLMs struggle in simple arithmetic tasks ( Nogueira et al. 2021 ; Gambardella et al. 2024 ) . Upon identifying such biases, we also explore a mitigation strategy by connecting recent work on contrastive decoding ( Li et al. 2023 ; O’Brien and Lewis 2023 ) and family-enhancement bias ( Goel et al. 2025 ) , aiming to cancel out similar score range biases encoded across the models from the same family. Our summarized contributions are as follows: • We first show that LLM judges have score range bias – a bias observed across different model sizes and families (Llama-3 and Qwen-2.5) when judging on direct assessment. • We then show that contrastive decoding, motivated by the observation of similar score range biases shared across models from the same family, successfully mitigates these biases. 2 Related Work We now review the related work on LLM judges focusing on the judge tasks and their biases. LLM Judge Tasks LLM judge tasks fall into two categories: direct or pointwise assessment ( Jones et al. 2024 ; Li et al. 2024 ; Zhu et al. 2025 ) and pairwise assessment ( Zheng et al. 2023 ; Ye et al. 2024 ) . Direct assessment ( Liu et al. 2023 ) involves LLM judges assigning numerical ratings to outputs without any other output references. In pairwise assessment, LLM judges show higher correlation with human preferences than direct assessment ( Liu et al. 2024 ) , supporting that the challenge remain in direct assessment, and we therefore focus on experimenting on direct assessment. LLM Judge Biases One known bias in LLM judges is self-enhancement bias—the tendency to favor their own output ( Liu et al. 2023 ; Zheng et al. 2023 ; Ye et al. 2024 ) even in proprietary models like GPT-4 ( Wataoka et al. 2024 ) . Extending beyond self-enhancement bias, Goel et al. 2025 reported a family enhancement bias where models favor outputs from the same model family. Assuming these family biases are bidirectional, we aim to cancel out using contrastive decoding. 3 Contrastive Decoding for Mitigating LLM Judge Biases Contrastive decoding ( Li et al. 2023 ) modifies the model outputs by using two models: a main model and an assistant model. Given the next token probability of a main model p_{\text{main}} and an assistant model p_{\text{asst}} , the final adjusted score is calculated by subtracting the weighted p_{\text{asst}} from p_{\text{main}} i.e., \log p_{\text{main}}-\lambda\log p_{\text{asst}} (1) where \lambda\in\mathbb{R} is the hyperparameter to control the magnitude of assistant model and logit e_{i} of token i is controlled by temperature t>0 i.e., p_{\text{Asst}}=\frac{e_{i}/t}{\sum_{j}e_{j}/t}. One difference we make compared to Li et al. 2023 is the inclusion of \lambda to further align the logit distribution between the two models motivated by our analysis in § 4.2 . 4 Experiments We focus on direct assessment on summarization since prior work reported that LLM judges fall short ( Ye et al. 2024 ) and they are commonly evaluated on summarization ( Panickssery et al. 2024 ) . 4.1 Setup Task and Metrics We focus on the summarization task where LLM judges are commonly used Liu et al. 2023 ; Panickssery et al. 2024 . The correlations between human annotations are measured using three metrics: Pearson, Spearman, and Kendall correlations. Score Scale and Ranges We use the 5 points Likert scale ( Likert 1932 ) on different score ranges (0-4, 1-5, 2-6, 3-7) 1 1 1 We stopped at 7 inspired by Likert 1932 showing high correlation between 5 points (1-5) and 7 points (1-7) results. . If output score parsing fails, we set to the lowest score following Liu et al. 2023 and if the parsed score exceeds the maximum, we clamp to the highest score in the range. Models We experiment on two model families. 2 2 2 We leave models like Prometheus ( Kim et al. 2024 ) specifically fineutuned on judge tasks as future work since multiple model sizes are not available for contrastive decoding and those models are finetuned towards 1-5 score range. For Llama-3 family ( Grattafiori et al. 2024 ) , we use Llama-3.1-8B-Instruct as the main model, Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct as the assistant model, and for Qwen2.5 family ( Qwen et al. 2025 ) , we use Qwen-2.5-14B-Instruct and Qwen-2.5-7B-Instruct as the main models, and Qwen-2.5-3B-Instruct as the assistant model. See Appendix A for the prompt used. (a) Llama-3 Family Results (b) Qwen2.5 Family Results Figure 2: Coherence score distribution in 2-6 score range with greedy decoding, contrastive decoding, and human annotations. The greedy decoding outputs from both Llama 8B and 3B models are highly skewed towards outputting score of 4 and Qwen2.5 3B and 7B models are outputting score of 2 showing similar biases are encoded in these models. Dataset We use SummEval ( Fabbri et al. 2021 ) , a summarization benchmark also used by Liu et al. 2023 which contains 100 news articles where each article is associated with 16 summaries with human annotation scores, which sums up to 1600 summaries. We use 10\% of the news articles

Read the complete text