Quantifying Language Disparities in Multilingual Large Language Models

Songbo Hu Ivan Vulić Anna Korhonen · 2025

Read Quantifying Language Disparities in Multilingual Large Language Models by Songbo Hu Ivan Vulić Anna Korhonen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17162.txt.

Introduction

Songbo Hu and co-authors give a clean way to measure gaps in how AI language models serve distinct languages. The work dates to 2025. Past large tests mix up many factors, such as the test set, the setup, and the model picked. That mix makes gaps hard to read and hard to trust.

The new frame splits those factors apart and gives three clear scores. One score shows how much of a language's reach a model taps, one shows spread, and one shows the reach itself. Tests on thirteen model forms and eleven data sets showed the frame is more sound. Top total scores did not mean fair play across languages.

Catalog record

CREATORSongbo Hu Ivan Vulić Anna Korhonen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17162

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantifying Language Disparities in Multilingual Large Language Models Songbo Hu Ivan Vulić Anna Korhonen Abstract Results reported in large-scale multilingual evaluations are often fragmented and confounded by factors such as target languages, differences in experimental setups, and model choices. We propose a framework that disentangles these confounding variables and introduces three interpretable metrics—the performance realisation ratio, its coefficient of variation, and language potential—enabling a finer-grained and more insightful quantification of actual performance disparities across both (i) models and (ii) languages. Through a case study of 13 model variants on 11 multilingual datasets, we demonstrate that our framework provides a more reliable measurement of model performance and language disparities, particularly for low-resource languages, which have so far proven challenging to evaluate. Importantly, our results reveal that higher overall model performance does not necessarily imply greater fairness across languages. 1 Introduction Contemporary NLP development relies on digital language datasets to build large language models (LLMs) and adapt them to downstream applications. Yet, none of these resources offers a balanced representation of the world’s more than 7,000 languages Blasi et al. 2022 ; Joshi et al. 2020 ; Bender et al. 2021 . This imbalance introduces two types of bias: intrinsic bias , which shapes the LLM’s internal representations; and adaptation bias , which affects downstream performance after task-specific adaptation Bommasani et al. 2021 . Given the common assumption of a strong correlation between the quantity of in-language data and LLM performance on that language, adaptation bias is often considered observable , as the amount of task-specific data is typically known; see, e.g., Hu et al. 2023 . In contrast, intrinsic bias in LLMs is more latent, becoming apparent only during evaluation on downstream tasks under parallel settings. Under the current paradigm, the evaluation of multilingual LLMs is often fragmented , in the sense that reported scores are typically obtained on isolated datasets, each covering different sets of languages and often targeting different NLP tasks. This leads to several limitations: (i) No existing dataset covers more than a small fraction of human languages, leading to severe and persistent under-representation. (ii) Reported scores are often confounded by multiple variables, such as the choice of languages, training and evaluation setups, making it difficult to compare results across languages, datasets, and metrics. (iii) Consequently, most findings from such multilingual evaluations remain qualitative, typically concluding that performance is uneven across languages, see e.g. Hu et al. 2020 ; Xuan et al. 2025 , without quantifying the magnitude of these disparities. (iv) Evaluating models on isolated datasets can lead to under- or over-estimation of performance for low-resource languages (see later in § 4 ), thereby hindering efforts to promote fairness across languages in NLP. To mitigate the aforementioned limitations, we propose a framework that disentangles confounding factors in multilingual evaluations, enabling the quantification of language disparities using evidence from multiple datasets. Drawing inspiration from measurement theory (§ 2 ), we define a latent construct called performance potential , which estimates the score a model is expected to achieve on a given task for a specific language, given the current state of NLP. Here, we adopt a norm-referenced approach (i.e., using the aggregate scores of the sampled population of LLMs as a reference). Then, we introduce the performance realisation ratio , defined as the ratio between the actual score achieved by the model and the estimated potential. Finally, we quantify model-level fairness using the coefficient of variation of this ratio across languages, which captures the relative dispersion in performance realisation across languages. Through a case study, we demonstrate that our proposed framework provides a more reliable measurement, particularly for low-resource languages. We showcase the core analysis code in Example 1 in the Appendix, and release the full code and data at: github.com/cambridgeltl/quantifying_language_disparities . 2 Related Work Measuring Performance as a Construct. Evaluation of language models can be viewed as a process of assigning numerical values to models, and thus naturally falls within the scope of measurement theory (see introductions in Hand 2016 ; Bandalos 2018 ). Measurement theory emphasises the distinction between a construct , the abstract concept we aim to measure, and the specific procedure used to produce its numerical representation. In our case, performance is the construct, while the reported scores serve as its representation. Tools from measurement theory have been increasingly adopted in NLP, including efforts to assess reliability Zhou et al. 2024 and robustness Hardy 2025 of LLMs, examine both the reliability and validity of NLG evaluation metrics Xiao et al. 2023 , reduce evaluation costs through adaptive testing Sedoc and Ungar 2020 , and analyse dataset difficulty Bachmann et al. 2024 ; Tambon et al. 2024 . Our framework draws inspiration from variance decomposition techniques used in generalisability theory Cronbach 1972 , which estimate the contributions of various measurement facets to observed score variability. We transpose and adapt this idea to multilingual evaluation. Definition of Fairness. Comparing the performance of multilingual LLMs can lead to a dilemma, where no single model consistently outperforms others across all languages. In such cases, aggregating multilingual performance into a single number for model selection inevitably challenges prevailing notions of fairness Choudhury and Deshpande 2021 . Similarly, cross-lingual comparisons also implicitly reflect on particular assumptions about fairness. Many multilingual NLP efforts aim to equalise performance across languages. This reflects a principle aligned with equal outcome . However, given the current disparities in digital resources across languages, such a goal is often difficult to operationalise. In this work, we adopt the broader view of equal opportunity , as discussed by Barocas et al. 2019 . Specifically, we define a fair LLM as one that is constructed in a way that enables all languages to realise their performance potential , that is, to achieve the level of performance that is realistically achievable given the current state of available resources. 3 Measurement Framework We propose a two-step framework: (i) estimating the performance potential for each language-task pair; and (ii) quantifying language disparities at both the language and model levels. Preliminaries. The input to our framework is a set of evaluation records, each represented as a tuple (\ell,t,m,s)\in\mathcal{D} , where \ell denotes a language, t denotes an evaluation task defined as a dataset–metric pair, m denotes a language model, and s\in\mathbb{R} is the observed performance score. Here, \ell , t , and m are treated as categorical variables. Step 1: Estimating Performance Potential. To isolate the achievable performance of a language from task difficulty and model capabilities, we model the observed score using a linear mixed-effects model as follows, which aligns with those widely used in generalisability theory Brennan 2001 : s_{\ell,t,m}=\mu+\alpha_{\ell}+\beta_{t}+u_{m}+\varepsilon_{\ell,t,m} (1) where \mu is the intercept; \alpha_{\ell} and \beta_{t} are the language and task-specific fixed effects, respectively; u_{m}\sim\mathcal{N}(0,\sigma^{2}_{u}) is a random effect for model m ; and \varepsilon_{\ell,t,m}\sim\mathcal{N}(0,\sigma^{2}) is the residual noise term. We then define the performance potential for a given language-task pair as the marginal mean score, averaged over the model population: \mathrm{PP}_{\ell,t}:=\widehat{\mathbb{E}}_{m}\left[s_{\ell,t,m}\right]=\mu+\alpha_{\ell}+\beta_{t} (2) where the expectation is taken over the random effect u_{m} with \mathbb{E}[u_{m}]=0 . Step 2: Quantifying Language Disparities. At the level of languages, we define the language potential as the performance of a typical model on a typical task, given current NLP resources: \mathrm{LP}_{\ell}:=\widehat{\mathbb{E}}_{t}\left[\mathrm{PP}_{\ell,t}\right]=\mu+\alpha_{\ell}+\operatorname{Mean}_{t}(\beta_{t}) (3)

Read the complete text