MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages

Dieuwke Hupkes, Nikolay Bogoychev * · 2025

Read MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages by Dieuwke Hupkes, Nikolay Bogoychev * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10356.txt.

Introduction

Dieuwke Hupkes and Nikolay Bogoychev present MultiLoKo, a benchmark that tests local knowledge in large language models across 31 languages. The main set holds 500 questions per language, each sourced to be locally relevant to that language. Two translated sets move human-authored questions between English and 30 other languages. Machine-authored translations are released for comparison, and the data splits into dev and blind test parts.

The team scored 11 base and chat models sold as multilingual. None did well: average scores were low and gaps between best and worst languages were large. Scores also shifted a lot with question language, which points to weak knowledge transfer across languages.

Catalog record

CREATORDieuwke Hupkes, Nikolay Bogoychev *
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10356

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages Dieuwke Hupkes, Nikolay Bogoychev * Abstract We present MultiLoKo, a new benchmark for evaluating multilinguality in LLMs covering 31 languages. MultiLoKo consists of three partitions: a main partition consisting of 500 questions per language, separately sourced to be locally relevant to the specific language, and two translated partitions, containing human-authored translations from 30 non-English languages to English and vice versa. For comparison, we also release corresponding machine-authored translations. The data is equally distributed over two splits: a dev split and a blind, out-of-distribution test split. MultiLoKo can be used to study a variety of questions regarding the multilinguality of LLMs as well as meta-questions about multilingual benchmark creation. We compute MultiLoKo scores for 11 base and chat models marketed to be multilingual and study their average performance, their performance parity across languages, how much their ability to answer questions depends on the question language, and which languages are most difficult. None of the models we studied performs well on MultiLoKo , as indicated by low average scores as well as large differences between the best and worst scoring languages. Furthermore, we find a substantial effect of the question language, indicating suboptimal knowledge transfer between languages . Lastly, we find that using local vs English-translated data can result in differences more than 20 points for the best performing models, drastically change the estimated difficulty of some languages . For using machines instead of human translations, we find a weaker effect on ordering of language difficulty, a larger difference in model rankings, and a substantial drop in estimated performance for all models. 1 Introduction With the growing presence and deployment of LLMs across the world, evaluating their abilities in languages other than English becomes more and more eminent. Yet, studying and evaluating multilinguality in LLMs remains a challenging enterprise, and it is hardly exaggerated to call the current state of multilingual evaluation in LLMs insufficient. Older multilingual benchmarks such as PAWS-X ( Zhang et al., 2019 ) , XNLI ( Conneau et al., 2018 ) or XCOPA ( Ponti et al., 2020 ) often do not fit the demands for evaluating auto-regressive models and are rarely used to evaluate recent models. Furthermore, their coverage of languages is relatively small compared to the number of languages in which LLMs are intended to be proficient. More often used for LLM evaluation are benchmarks translated from English, such as MGSM ( Shi et al., 2023 , translated GSM8K,) , MMMLU ( OpenAI, 2025 , tranlated MMLU,) or (less frequently) Belebele ( Bandarkar et al., 2024 ) . These benchmarks provide good coverage over many languages, but using translated data comes with its own set of issues. One such issues is that even when human- rather than machine-authored translations are used, translated data is known to differ from native text in several ways ( Clark et al., 2020 ) . Furthermore, using translated benchmarks imposes a strong English-centric bias: translated data may be multilingual on the surface, it is not in its content. The benchmarks MLQA ( Lewis et al., 2020 ) and TidyQA ( Clark et al., 2020 ) to some extent address the issue by sourcing data separately for different languages. Even in their sourcing protocols, however, there is no explicit focus on selecting locally relevant content for the chosen languages. 1 1 1 An exception to this is the benchmark EXAMS ( Hardalov et al., 2020 ) , which consists of exams separately sourced for each language. For reasons unknown to the authors of this work, it was never used for any prominent LLM release, with the exception of ( Dubey et al., 2024 ) , who deployed it for training rather than evaluation. In addition to that, their coverage is again small compared to the above mentioned translated benchmarks. In response to these issues, we introduce a wide-coverage multilingual benchmark with locally-sourced questions for 31 different languages. Because the benchmark targets multilingual local knowledge, we dub it MultiLoKo . The release of MultiLoKo serves two interconnected goals: 1) Provide a better means to evaluate multilinguality in LLMs; 2) Provide data to study the effect of various design choices in multilingual evaluation. To address our first goal, we create 500 questions per language, written from scratch for each language, using a sourcing protocol specifically designed to ensure local relevance of the question topics. To also reap the benefits of parallel data, we commissioned both human and machine-authored translations for all non-English questions into English and vice versa, providing a total of 15500 parallel questions, sourced across the 31 languages in the benchmark. The translated data facilitates the study of transfer between languages and also serves our second goal. By comparing the English-translated data with the locally sourced data, we can explicitly compare the adequacy of using translated benchmarks; by comparing human- with machine-authored translations, we can better estimate the potential issues of the latter. To prevent quick overfitting and inadvertent contamination, we release a development set of the benchmark, while test scores can only be obtained through an external provider. 2 2 2 The MultiLoKo data, five few-shot examples per language, an evaluation script, a set of language-specific prompts, and information about test-score submissions can be found at https://github.com/facebookresearch/multiloko/ . We provide elaborate analyses for both our goals. We compute average performance and language parity scores on the locally sourced data for 11 models marketed for their multilinguality ( § 5.1 ); we investigate whether these models exhibit knowledge transfer between different languages ( § 5.2 ); we study the impact of local sourcing versus translating on model rankings and language difficulty ( § 5.4.1 ); we analyse the difficulty of the included languages through various lenses ( § 5.3 ); and we conduct an analysis into the difference between human- and machine-authored translation ( § 5.4.3 ). We find that, of the models we consider, the best performing model is Gemini 2.0 Flash, with an average performance of 34.4 points, and an almost 35 point gap between the best and the worst language. Llama 3.1 405B and GPT4-o are close contenders in terms of average scores (34.3 and 34.0, respectively), but both have substantially higher language gaps (39 and 49 points). Almost across the board, model performances are better when questions are asked in the language to which the content is relevant, indicating suboptimal knowledge transfer between languages, a result that is mirrored by low response-consistency across question language. Next, we study the relevance of using locally sourced data as opposed to translated English data as well as whether it matters if translations are authored by humans or machines. We find that the estimated difficulty of some languages changes drastically across the two sourcing setups, within the range of 15 points decrease and 8 points increase on average across models. The rank correlation between average language difficulty score is 0.78. Furthermore, individual model scores between locally and English-translated data can differ up to 22 points for some languages. However, changing the sourcing setup does not impact model rankings, suggesting that using translated data may be suitable for comparing models but less for model development or language prioritisation. For using machine- instead

Read the complete text