Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck

Hongbin Zhang, Kehai Chen, Xuefeng Bai, Youcheng Pan, Yang Xiang, Jinpeng Wang, Min Zhang · 2026

Read Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck by Hongbin Zhang, Kehai Chen, Xuefeng Bai, Youcheng Pan, Yang Xiang, Jinpeng Wang, Min Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10351.txt.

Introduction

Hongbin Zhang and co-authors tackle translationese bias, where models used as multilingual judges favor machine translated text over human writing, most in low resource languages. They trace the bias to spurious links with English shaped latent structure and cross lingual predictability. Their DIBJudge framework fine tunes a model to keep only judgment critical features, pushing spurious factors into a separate bias branch, with a penalty keeping the two apart. On reward benchmarks and a dedicated bias suite, DIBJudge beats strong baselines and cuts the bias a lot.

Catalog record

CREATORHongbin Zhang, Kehai Chen, Xuefeng Bai, Youcheng Pan, Yang Xiang, Jinpeng Wang, Min Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95976
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10351

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck Hongbin Zhang, Kehai Chen, Xuefeng Bai, Youcheng Pan, Yang Xiang, Jinpeng Wang, Min Zhang Abstract Large language models (LLMs) have become a standard for multilingual evaluation, yet they exhibit a severe systematic “ translationese bias ”. In this paper, “translationese bias” is characterized as LLMs systematically favoring machine-translated text over human-authored references, particularly in low-resource languages. We attribute this bias to spurious correlations with (i) latent manifold alignment with English and (ii) cross-lingual predictability. To mitigate this bias, we propose DIBJudge , a robust fine-tuning framework that learns a minimally sufficient, judgment-critical representation via variational information compression, while explicitly isolating spurious factors into the dedicated bias branch. Furthermore, we incorporate a cross-covariance penalty that explicitly suppresses statistical dependence between robust and bias representations, thereby encouraging effective disentanglement. Extensive evaluations on multilingual reward modeling benchmarks and a dedicated translationese bias evaluation suite demonstrate that the proposed DIBJudge consistently outperforms strong baselines and substantially mitigates translationese bias. 1 Introduction The emergence of Large Language Models (LLMs) has revolutionized evaluation paradigms ( 22 ; 44 ) , establishing “LLM-as-a-Judge” as a standard framework for multilingual assessment ( 63 ; 54 ; 3 ; 26 ; 18 ; 17 ) . Consequently, ensuring the accuracy and robustness of these automated judges across diverse languages has become a critical necessity ( 52 ; 7 ) . Figure 1 : Translationese Bias Severity of GPT-4o across languages. Languages are sorted by resource availability from low (top) to high (bottom). The trend line illustrates the inverse relationship between resource availability and translationese bias. However, the reliability of LLM judges is frequently undermined by systematic biases ( 71 ; 80 ; 19 ) , such as position ( 60 ; 70 ) and verbosity bias ( 59 ) . While these limitations are well-studied in English contexts ( 8 ; 87 ) , specific failure modes within multilingual settings remain underexplored. In this paper, we characterize a distinct bias of LLM-as-a-Judge in multilingual contexts, termed translationese bias , in which LLMs favor machine-translated content over human-authored reference, even when the former is semantically flawed. To investigate this bias, we first conduct a comprehensive evaluation across a diverse spectrum of languages. As shown in Figure 1 , this bias is not only pervasive but is significantly exacerbated in low-resource languages. Crucially, our further attribution analysis suggests that LLM judges may conflate generation quality with two potential spurious factors: (a) latent manifold alignment with English, and (b) cross-lingual predictability. While recent advancements in multilingual LLM judges have yielded promising results ( 54 ; 3 ; 84 ) , most existing methods remain grounded in standard Supervised Fine-Tuning (SFT). However, SFT is susceptible to exploiting spurious correlations ( 61 ; 23 ; 11 ) , thereby limiting its efficacy in mitigating translationese bias. To this end, we propose the Disentangled Information Bottleneck Judge ( DIBJudge ), a robust fine-tuning framework that explicitly decouples the latent representation into two components: a robust representation that preserves judgment-critical semantic information, and a bias representation that isolates the spurious factors identified above (i.e., latent manifold alignment with English and cross-lingual predictability). We leverage variational information compression to learn a robust, minimally sufficient representation that preserves only information essential for accurate judgment. To further encourage disentanglement between robust and bias representation, we penalize their mutual dependence during training. Extensive experiments on multilingual reward modeling benchmarks, including M-RewardBench ( 25 ) and MM-Eval ( 63 ) , demonstrate that DIBJudge consistently outperforms strong baselines, yielding improved multilingual reward modeling performance. Moreover, evaluations on a dedicated translationese bias suite confirm that DIBJudge substantially mitigates the severity of translationese bias. In summary, we make three key claims: (i) we characterize translationese bias in multilingual LLM judges and identify two related spurious factors—latent manifold alignment with English and cross-lingual predictability, (ii) we propose DIBJudge , a robust fine-tuning framework that disentangles judgment-critical semantics from spurious factors, and (iii) we show that DIBJudge consistently outperforms strong baselines on multilingual reward modeling benchmarks while effectively mitigating translationese bias. 1 1 1 Our anonymous code repo is here. 2 Preliminary Analysis of Translationese Bias To systematically study translationese bias, we structure our preliminary analysis around two research questions: (i) RQ1: How does translationese bias vary across languages with different levels of resource availability? (ii) RQ2: What kinds of spurious factors are associated with this bias? Bias Evaluation Protocol. We construct a controlled translationese bias benchmark derived from BELEBELE ( 5 ) , a multilingual reading comprehension dataset spanning 122 languages. Following the language-resource taxonomy of 33 , we stratify languages into high-, medium-, and low-resource tiers and sample 10 representative languages per tier. For each language, we evaluate on 200 instances, yielding a balanced benchmark across resource levels. The full list of selected languages is provided in Appendix A.1 . We formulate translationese bias evaluation as a pairwise preference task, where an LLM judge compares two candidate responses for the same query: (i) a human-authored reference x_{H} , and (ii) a machine-generated variant x_{M} obtained via back-translation to induce translationese artifacts. To avoid position bias, each instance i is evaluated under both forwarding and reverse ordering. Details are available in Appendix A.2 . Bias Metric Definition. Let y_{i}\in\{0,1\} indicate whether the judge prefers x_{M} in the forward order, and let \bar{y}_{i}\in\{0,1\} denote the corresponding preference in the reverse order. We define bias severity \mathcal{S}_{\mathrm{bias}} as the fraction of consistent judgments that favor the machine-generated output: \mathcal{S}_{\mathrm{bias}}=\frac{\sum_{i=1}^{N}\mathbb{I}\left[y_{i}=1\land\bar{y}_{i}=1\right]}{\sum_{i=1}^{N}\mathbb{I}\left[y_{i}=\bar{y}_{i}\right]}. (1) (a) Machine Win Rate vs. CAD (b) SSR Density (Human vs. Machine Win) (c) ROC Analysis of CAD and SSR Figure 2 : Correlation analysis of judge preference with confounding factors. (a) Machine win rate decreases monotonically as CAD increases, indicating that judge preference spuriously tracks latent manifold isomorphism with English. (b) SSR distributions exhibit a clear drift between human-win and machine-win cases, showing that the judge systematically favors higher-likelihood outputs. (c) ROC curves confirm that both CAD and SSR reliably predict judge outcomes, reinforcing the attribution that translationese bias is mediated by latent manifold isomorphism with English and high predicative confidence. 2.1 Quantifying Bias across Resource Levels (RQ1) Figure 1 illustrates the translationese bias severity of GPT-4o across varying language resource levels. We observe two salient patterns: first, translationese bias is pervasive across the entire linguistic spectrum; second, there is a distinct inverse correlation between resource availability and the magnitude of bias. Specifically, while high-resource languages exhibit minimal bias, low-resource languages demonstrate significantly elevated severity. These findings expose translationese bias as a severe yet previously neglected failure mode in multilingual LLM judges, which critically undermines evaluation reliability and disproportionately compromises under-resourced languages. 2.2 Attribution Analysis of Translationese Bias (RQ2) Due to the scarcity of high-quality native resources ( 55 ; 30 ) , multilingual LLMs are typically pre-trained on English-dominated corpora ( 39 ; 77 ) and subsequently adapted using translated or synthetic data ( 51 ; 82 ) . Accordingly, we hypothesize that translationese bias stems from spurious correlations induced across these two stages: (i) latent manifold alignment with English, where non-English representations are implicitly aligned to an English-centric latent space during pre-training; and (ii) cross-lingual predictability, where the judge over-relies on probability heuristics

Read the complete text