Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study
Irune Zubiaga, Aitor Soroa, Rodrigo Agerri · 2026
Read Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study by Irune Zubiaga, Aitor Soroa, Rodrigo Agerri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28710.txt.
Introduction
Irune Zubiaga, Aitor Soroa, and Rodrigo Agerri study multilingual LLMs used as judges of generated text. Most prior work covers English only, so they test English, Spanish, and Basque as high-, mid-, and low-resource languages. They compare instruction translation, monolingual versus multilingual training, and model size, extending two meta-evaluation datasets to Basque and Spanish. With in-domain data, small fine-tuned models rival proprietary models, while larger zero-shot models win out of domain, and out-of-domain fine-tuning can hurt.
Catalog record
| CREATOR | Irune Zubiaga, Aitor Soroa, Rodrigo Agerri |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45065 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28710
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study Irune Zubiaga, Aitor Soroa, Rodrigo Agerri Abstract Large language models (LLMs) are increasingly used for the automatic evaluation of generated text, yet most prior work focuses on English. Despite the growing demand for multilingual evaluation, extending LLM-based evaluators to multilingual settings remains challenging, particularly for low-resource languages and scenarios where in-domain data is scarce. This work explores several strategies for developing multilingual LLMs-as-a-judge, considering whether in-domain data is available for fine-tuning or not. We systematically analyze English, Spanish, and Basque, representing high-, mid-, and low-resource languages, considering instruction translation, monolingual versus multilingual supervision, and model size. For evaluation, we extend two existing meta-evaluation datasets to Basque and Spanish. Our results reveal key trade-offs: When in-domain data is available, fine-tuned smaller models can achieve performance comparable to proprietary models, whereas zero-shot evaluation with larger models proves more effective in out-of-domain settings. We also observe that fine-tuning on out-of-domain data can adversely affect model performance. These findings provide practical guidance for building efficient, reliable multilingual evaluation pipelines. The data and code are publicly available at hitz-zentroa/mJudge . 1 Introduction Large language models (LLMs) are increasingly used not only for text generation but also for evaluating the outputs of other LLMs. However, most existing research remains centered on English, leaving open research questions unanswered about how to extend LLM-based evaluation to multilingual contexts, especially when low-resource languages are involved. 1 1 1 We use low-resource to denote limited representation and performance in current open-weight language models, rather than overall linguistic vitality ( 20 ) . In this work, we study how to design effective training and evaluation strategies that enable robust multilingual performance across languages with different levels of support in existing pretraining corpora. We focus on three languages with varying resource availability: English (high-resource), Spanish (mid-resource), and Basque (low-resource). Typologically, they span distinct families and structures: English is Germanic, Spanish is Romance, and Basque is a language isolate with agglutinative morphology. These languages provide a representative and controlled testbed for analyzing how evaluation strategies behave across multilingual conditions, while capturing complementary typological and resource-level variation in a computationally tractable setting. We frame our investigation around two conditions: (i) the availability of in-domain data for fine-tuning the evaluators, and (ii) the absence of in-domain training data. Each scenario raises a distinct set of evaluation questions. When training data is available, we examine how translating the data from English to the target language affects performance, the relative benefits of cross-lingual transfer versus multilingual training, and the impact of model size. We also investigate whether it is more effective to train a model directly on target-language data or to leverage a model already proficient in said language and adapt it to the task. Conversely, in the absence of in-domain data, we explore whether fine-tuning on related out-of-domain data is helpful and how model size influences zero-shot generalization. This work makes several key contributions. First, in settings with in-domain data, we find that smaller 8B models remain highly competitive with much larger 70B and proprietary models, suggesting that scale alone is not the decisive factor when task-specific data is available. Second, we observe that multilingual training improves performance in Spanish and Basque compared to monolingual training, indicating effective cross-lingual transfer for evaluation tasks. We further find that providing evaluation rubrics and instructions in English, rather than translating them into the target language, yields more consistent judgments across languages, suggesting that full prompt translation is often unnecessary. Third, we find that starting from a language-proficient model and adapting it to the evaluation task is generally more effective than training a model on target-language task-specific data. Finally, in out-of-domain settings, we find that fine-tuning on related data can degrade performance in larger models, while smaller models benefit but still underperform compared to zero-shot large models. 2 Related Work As the use of LLMs continues to grow, the need for multilingual capabilities becomes increasingly important to ensure accessibility for a diverse range of users. While most LLMs remain English-centric, new models targeting other languages have started to emerge. For instance, Pangea ( 33 ) is a fully open multilingual and multimodal LLM trained on diverse instructions in 39 languages, designed to improve cross - cultural representation and evaluation coverage. Similarly, EuroLLM ( 18 ) develops a suite of open - weight multilingual models supporting all official EU languages, demonstrating strong performance on generation and translation benchmarks. Efforts have also focused on broader multilingual coverage, including low-resource languages, such as BLOOM: A 176B-Parameter Open-Access Multilingual Language Model ( 29 ) . These initiatives highlight the importance of developing and evaluating models that cater to linguistically diverse populations. Multilingual pre-training can improve cross-lingual transfer to low-resource languages, although gains tend to saturate as model capacity is shared across many languages, a phenomenon known as the curse of multilinguality ( 4 ; 2 ) . Cross-lingual transfer helps mitigate data scarcity ( 36 ; 5 ) , but fine-tuning on additional languages may introduce interference or catastrophic forgetting ( 9 ) . To address this, prior work explores selecting related source languages and parameter-efficient methods such as adapter-based approaches, which can improve transfer in low-resource settings ( 1 ; 22 ) . Due to the poor correlation of traditionally used metrics such as BLEU, ROUGE-L or BertScore with human judgments ( 27 ) , LLMs are increasingly being used as evaluators, achieving considerably higher correlations ( 34 ) . Unfortunately, most of this work relies on proprietary LLMs ( 19 ) , which limits reproducibility and accessibility. Some open alternatives have emerged, such as JudgeLM ( 35 ) or Prometheus ( 15 ) . However, most of this research has been conducted primarily in English. While a few works such as M-Prometheus ( 23 ) have begun exploring multilingual settings, determining the best method for training a multilingual LLM evaluator remains an open and largely unexplored research question. While some efforts have been made in evaluating these systems, benchmarks remain scarce. A large portion of existing benchmarks are automatically generated using GPT or similar LLMs. While such benchmarks are relatively easy to scale, they can introduce model bias, circularity, and lack of robustness, since the same families of models are often used for both generation and evaluation ( 17 ) . On the other hand, human-annotated benchmarks are more reliable but also more costly and difficult to construct. A considerable amount of these rely on preference-based annotations (e.g., pairwise comparisons between outputs), rather than absolute scoring ( 11 ) . However, prior work shows that pairwise evaluation can amplify biases in LLM-as-a-judge, such as favoring verbosity or authoritative tone, whereas pointwise evaluation, which assesses outputs independently, is less susceptible to these issues ( 13 ) . Nonetheless, some human-scored resources for absolute evaluation are available for English, such as FLASK ( 32 ) , highlighting the potential value of expanding this line of work. However, very few available multilingual benchmarks are currently available. Among them, RECON ( 7 ) stands out as a human-supervised, multilingual evaluation dataset. It contains prompts written in multiple languages paired with English references and evaluation rubrics. However, the currently covered languages (Bengali, German, French, Hindi, Telugu, and Urdu) remain limited.