Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History
Yevhen Kostiuk · 2025
Read Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History by Yevhen Kostiuk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09154.txt.
Introduction
Yevhen Kostiuk and three co-authors test how well large language models know the history of Lithuania and of the world, in many tongues at once. Their paper is from two thousand twenty-five.
The tasks are multiple-choice. The asks were turned into Baltic, Nordic, and other tongues, to see if facts flow from kin cultures. Ten-plus models are tried, from a top closed one to small open ones, plus models tuned for Nordic tongues. The top closed model won in all tongue groups, with a slight edge for Baltic and Nordic ones. Big open models did well but tied less to Baltic tongues. Small models showed clear gaps.
Catalog record
| CREATOR | Yevhen Kostiuk |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09154
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History Yevhen Kostiuk Abstract In this work, we evaluated Lithuanian and general history knowledge of multilingual Large Language Models (LLMs) on a multiple-choice question-answering task. The models were tested on a dataset of Lithuanian national and general history questions translated into Baltic, Nordic, and other languages (English, Ukrainian, Arabic) to assess the knowledge sharing from culturally and historically connected groups. We evaluated GPT-4o, LLaMa3.1 8b and 70b, QWEN2.5 7b and 72b, Mistral Nemo 12b, LLaMa3 8b, Mistral 7b, LLaMa3.2 3b, and Nordic fine-tuned models (GPT-SW3 and LLaMa3 8b). Our results show that GPT-4o consistently outperformed all other models across language groups, with slightly better results for Baltic and Nordic languages. Larger open-source models like QWEN2.5 72b and LLaMa3.1 70b performed well but showed weaker alignment with Baltic languages. Smaller models (Mistral Nemo 12b, LLaMa3.2 3b, QWEN 7B, LLaMa3.1 8B, and LLaMa3 8b) demonstrated gaps with LT-related alignment with Baltic languages while performing better on Nordic and other languages. The Nordic fine-tuned models did not surpass multilingual models, indicating that shared cultural or historical context alone does not guarantee better performance. 1 Introduction Large Language Models (LLMs) provide a functional framework for tackling various natural language processing (NLP) tasks, such as question-answering ( Izacard et al. 2022 ; Dong et al. 2024 ) , machine translation ( Zhu et al. 2023 ; Kocmi et al. 2024 ) and so on. However, LLMs have shown less reliable results for low-resource languages ( Ranjan et al. 2024 ; Sakib and Das 2024 ) due to the smaller fraction of available data in comparison to English and a few other widely spoken languages. Benchmarking multilingual LLMs across languages is essential for evaluating their capabilities. However, the availability of high-quality, culturally aligned datasets remains a challenge. This need for culturally aligned high-quality datasets becomes even more critical when evaluating historical knowledge, where ensuring linguistic and cultural fairness adds a layer of complexity. Verifying comparability of the results on historical knowledge QA datasets requires that a single set of historical events is queried in all languages. The choice of that set is likely to be biased to events more represented in widely spoken languages. Conversely, events that are more region- or cultural-specific are less likely to occur in the benchmarks. Finding and addressing these gaps is important to improve the fairness of LLMs and highlight historical and cultural biases. In this work, we focus on evaluating multilingual LLMs on Lithuanian and general history. Our goal is to determine how LLMs perform on Lithuanian history exam questions when prompted in different languages and explore the alignment between languages and historical awareness, particularly within the Nordic and Baltic language groups. Our contribution is the following: • We automatically translated publicly available Lithuanian history exam question-answering dataset into Nordic (Danish, Finnish, and Swedish), Baltic (Estonian and Latvian), and other (Arabic, Ukrainian, and English) languages and partially manually evaluated it. • We tested GPT-4o ( OpenAI et al. 2024 ) , LLaMa3.2 3b, LLaMa3 8b, LLaMa3.1 8b and 70b ( Dubey et al. 2024 ) , Mistral Nemo 12b ( Jiang et al. 2023 ) , QWEN2.5 7b and 72b Team 2024 ; Yang et al. 2024 , and GPT-SW3 and Nordic-trained LLaMa3 8b ( Ekgren et al. 2023 ) models and compared their achieved accuracy scores per language and its average per language group. Our findings revealed that GPT-4o consistently outperformed other models across all evaluated languages and language groups on a dataset of LT-related and general history questions. Larger open-source models, such as LLaMa3.1 70b and QWEN2.5 72b, also demonstrated strong and consistent performance in all languages. In contrast, smaller models like Mistral Nemo 12b, LLaMa3 8b, LLaMa3.2 3b, and LLaMa3.1 8b showed notable gaps in their historical knowledge from a Lithuanian perspective, particularly with Baltic languages, despite Lithuanian being part of this group. The best performance was observed in the Nordic language group, suggesting that cultural or historical alignment alone does not ensure higher accuracy. Interestingly, the Nordic pre-trained models failed to surpass the multilingual model. 2 Related Work Pre-trained LLMs have exhibited a remarkable ability to encode and retrieve factual and common knowledge across different languages ( Wang et al. 2023 ; Zhao et al. 2024 ) . However, there is a notable variation in model performance across languages, with a strong shift toward high-resource languages ( Qi et al. 2023 ) , particularly languages with Latin scripts ( Ifergan et al. 2024 ) . The datasets used for benchmarking multilingual LLMs are created using either one of the two approaches: human annotation ( Kocmi et al. 2023 ; Goyal et al. 2022 ) or translating existing annotated datasets using LLMs ( Lai et al. 2023 ) . Although datasets created by human annotators provide accurate translations and task-specific precision, they require considerable investment of both time and finances ( Yang et al. 2019 ) . On the other hand, with an advancement of LLMs, the translation performance of automatic tools has been significantly boosted lately. For example, ChatGPT demonstrates fewer errors with the launch of the GPT-4 engine, even for distant languages ( Jiao et al. 2023 ) . The quality control research on the DeepL translation tool found that DeepL 1 1 1 https://www.deepl.com/ performed well in terms of translation accuracy, fluency, and naturalness, reaching an overall semantic similarity score 94.13 ( Linlin 2024 ) . This improvement elevated the creation of benchmark datasets on various tasks. DeepL was used for creating the X-FACT multilingual factual knowledge dataset translated in 25 languages ( Gupta and Srikumar 2021 ) . In the research ( Thellmann et al. 2024 ) , five well-known datasets of various tasks were translated by DeepL into 21 European languages. LLMs with different numbers of parameters were evaluated on newly introduced datasets. The authors observed that models generally achieve higher performance on Romance and Germanic languages compared to Slavic languages. ChatGPT was utilized to translate the 158K English instructions into 26 languages, including 7 low-resource languages ( Lai et al. 2023 ) . The data was used to instruction-tune LLM for multiple languages using reinforcement learning from human feedback. The resulting framework, Okapi, was also evaluated on datasets translated by ChatGPT from English into 26 selected languages. Relying on the aforementioned approaches, in our work we utilized ChatGPT and DeepL to create a dataset with special consideration on Baltic and Nordic languages. To ensure the quality of translation for each language, 100 samples of English version and it’s corresponding translation were manually evaluated by native speakers. On average, 77% of such language pairs were approved. We offer evaluation on Lithuanian history exam tasks, assessing the ability of multilingual LLMs to exhibit particular cultural and historical knowledge and evaluating the alignment of this knowledge within regional language groups. 3 Methodology In this paper, we investigate performance consistency of LLMs within Nordic and Baltic language groups on the Lithuanian history exams questions. We hypothesized that the LLMs perform better in this domain, when presented with questions in languages from Nordic and Baltic groups than from other due to the cultural, linguistic and historical similarities. Figure 1: Example of the dataset