Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge
Xiaolin Zhou, Zheng Luo, Yicheng Gao, Qixuan Chen, Xiyang Hu, Yue Zhao, Ruishan Liu · 2026
Read Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge by Xiaolin Zhou, Zheng Luo, Yicheng Gao, Qixuan Chen, Xiyang Hu, Yue Zhao, Ruishan Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13649.txt.
Introduction
Xiaolin Zhou and co-authors test whether AI judges favor some languages over others in 2026. In LLM-as-a-judge setups, a model grades text quality for a given task. Past work shows such judges can be biased by traits of the text. Those biases may not match what human readers prefer.
The team asks if fluency, smooth and easy prose, can crowd out fairness. A judge might rate slick text high and plain text low for the wrong reasons. Their study probes language bias in pair-wise judging, where two texts face off. Clear proof of bias would push for fairer ways to grade with models.
Catalog record
| CREATOR | Xiaolin Zhou, Zheng Luo, Yicheng Gao, Qixuan Chen, Xiyang Hu, Yue Zhao, Ruishan Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13649
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge Xiaolin Zhou, Zheng Luo, Yicheng Gao, Qixuan Chen, Xiyang Hu, Yue Zhao, Ruishan Liu Abstract Recent advances in Large Language Models (LLMs) have incentivized the development of LLM-as-a-judge, an application of LLMs where they are used as judges to decide the quality of a certain piece of text given a certain context. However, previous studies have demonstrated that LLM-as-a-judge can be biased towards different aspects of the judged texts, which often do not align with human preference. One of the identified biases is language bias , which indicates that the decision of LLM-as-a-judge can differ based on the language of the judged texts. In this paper, we study two types of language bias in pairwise LLM-as-a-judge: (1) performance disparity between languages when the judge is prompted to compare options from the same language, and (2) bias towards options written in major languages when the judge is prompted to compare options of two different languages. We find that for same-language judging, there exist significant performance disparities across language families, with European languages consistently outperforming African languages, and this bias is more pronounced in culturally-related subjects. For inter-language judging, we observe that most models favor English answers, and that this preference is influenced more by answer language than question language. Finally, we investigate whether language bias is in fact caused by low-perplexity bias, a previously identified bias of LLM-as-a-judge, and we find that while perplexity is slightly correlated with language bias, language bias cannot be fully explained by perplexity only. 1 Introduction Figure 1: Illustration of language bias in the same-language judging and inter-language judging scenario. Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks. One of the important applications of LLMs is LLM-as-a-judge. In this framework, LLMs are prompted to assess the quality of text inputs, either by assigning scores (pointwise), selecting the better option from a pair (pairwise), or ranking multiple candidates (listwise) ( Li et al. 2024 ) . Pairwise LLM-as-a-judge has attracted particular attention due to its natural alignment with applications such as automatic model evaluation ( Li et al. 2023b ) and reinforcement learning from AI feedback ( Lee et al. 2024 ) . For these applications to be trustworthy, however, the judge model must provide fair and unbiased decisions that align well with human preferences. Unfortunately, prior research has revealed that LLM judges exhibit multiple types of systematic biases that compromise their reliability ( Zheng et al. 2023 ; Ye et al. 2024 ; Wang et al. 2024 ; Shi et al. 2025 ) . One of them is language bias, where LLMs are demonstrated to show undesired preferences when given texts written in different languages. Specifically, several recent works ( Fu and Liu 2025 ; Doddapaneni et al. 2025 ; Padarha et al. 2025 ) have revealed that in a pointwise judging scenario, LLMs can assign higher scores to answers written in higher-resource languages. Despite growing awareness of language bias, many critical questions remain unanswered. For example, first, existing studies have primarily examined pointwise judging; the extent of language bias in pairwise comparison settings, where two options are directly compared, remains unclear. Second, it is unknown whether language bias is uniform across domains or more pronounced in subjects tied to cultural and linguistic contexts (such as history or social sciences) compared with language-agnostic domains like mathematics or physics. Third, it is yet unclear whether language bias can be attributed to low-perplexity bias ( Panickssery et al. 2024 ; Wataoka et al. 2025 ) , a previously identified bias of LLM-as-a-judge to favor answers with low perplexity. In other words, is language bias simply a special case where options in high-resource languages happen to have lower perplexity, and thus undesirably preferred by the judge model? In this paper, we present a comprehensive investigation of language bias in pairwise LLM-as-a-judge. We study two distinct scenarios, shown in Figure 1 : same-language judging, where both candidate answers are written in the same language, and inter-language judging, where the two answers are presented in different languages. For same-language judging, we examine performance disparities across languages. For inter-language judging, we investigate whether models exhibit systematic preferences for answers written in high-resource languages. For both settings, we also study the factors that affect the extent of language bias. Finally, we conduct an analysis to determine whether language bias can be reduced to low-perplexity bias. 2 Related Works 2.1 LLM-as-a-Judge Most previous works ( Li et al. 2024 ) apply LLM-as-a-judge in three different forms: pointwise judging, pairwise judging, and listwise judging. In this paper, we focus on pairwise judging, which is the process where the judge LLM is prompted to choose a better one from two candidate text inputs. Pairwise judging has drawn significant attention recently for its potential applications such as automatic model evaluation ( Li et al. 2023b ; Zhao et al. 2025 ) , learning from AI feedback ( Lee et al. 2024 ; Xu et al. 2024 ) , etc. A bottleneck of the application of pairwise LLMs to broader application lies in their performance, often measured by their correlation with human preference. In light of this, some previous works proposed expert models for LLM-as-a-judge, such as Auto-J ( Li et al. 2023a ) , JudgeLM ( Zhu et al. 2025 ) , Prometheus ( Kim et al. 2024a ) , Prometheus-2 ( Kim et al. 2024b ) , and multilingual expert judge model M-Prometheus ( Pombal et al. 2025 ) . Other previous works analyze the cause of mismatch with human preferences, and identify certain biases commonly observed in LLM-as-a-judge. 2.2 Biases of LLM-as-a-Judge Previous studies have identified a series of biases in LLM-as-a-judge, which limit the accuracy and potential application scope of the technique. For example, position bias ( Wang et al. 2024 ) is a well known issue in pairwise LLM-as-a-judge, where the judge LLM tends to unproportionately favor the option at a certain position. One line of previous research studies self-preference bias ( Panickssery et al. 2024 ) , which shows that LLMs tends to unfairly favor options that are generated by themselves. Another previous work ( Wataoka et al. 2025 ) show that the favor of the models’ own generations can be potentially generalized to the favor of low-perplexity options over high-perplexity ones for the judge LLM. This indicates that judge LLMs might favor the responses that they are more “familiar” with rather than the ones that are factually better. Another type of bias in LLM-as-a-judge is language bias . It has been shown that in a pointwise LLM-as-a-judge setting where the judge model is prompted to assign a score to a given input, the judge model tends to assign lower scores to inputs written in low-resource languages ( Fu and Liu 2025 ; Doddapaneni et al. 2025 ; Padarha et al. 2025 ; Faisal et al. 2025 ) . However, a lot of questions remain unclear. For example, (1) whether this phenomenon generalizes to pairwise LLM-as-a-judge, (2) is this phenomenon more significant in specific settings (e.g. in texts related to social or cultural backgrounds compared with texts that are cultural-agnostic such as