Evaluating Polish linguistic and cultural competency in large language models

Sławomir Dadas, Małgorzata Grębowiec, Michał Perełkiewicz, Rafał Poświata · 2025

Read Evaluating Polish linguistic and cultural competency in large language models by Sławomir Dadas, Małgorzata Grębowiec, Michał Perełkiewicz, Rafał Poświata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00995.txt.

Introduction

Sławomir Dadas and three co-authors present a study from twenty twenty-five on how well language models know Polish tongue and culture. Skill at text in many tongues does not prove grasp of a culture. Facts of daily life, past events, tales, books, and pop works all feed true fluency. Misses there breed subtle errors that are hard to spot.

Their test set holds six hundred hand made questions in six groups. Groups span past events, land facts, customs, arts, grammar, and word stock. Over thirty models, open and closed, took the test. The study gives a fresh view of Polish skill in these models, past plain language tasks.

Catalog record

CREATORSławomir Dadas, Małgorzata Grębowiec, Michał Perełkiewicz, Rafał Poświata
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-00995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Polish linguistic and cultural competency in large language models Sławomir Dadas, Małgorzata Grębowiec, Michał Perełkiewicz, Rafał Poświata Abstract Large language models (LLMs) are becoming increasingly proficient in processing and generating multilingual texts, which allows them to address real-world problems more effectively. However, language understanding is a far more complex issue that goes beyond simple text analysis. It requires familiarity with cultural context, including references to everyday life, historical events, traditions, folklore, literature, and pop culture. A lack of such knowledge can lead to misinterpretations and subtle, hard-to-detect errors. To examine language models’ knowledge of the Polish cultural context, we introduce the Polish linguistic and cultural competency benchmark, consisting of 600 manually crafted questions. The benchmark is divided into six categories: history, geography, culture & tradition, art & entertainment, grammar, and vocabulary. As part of our study, we conduct an extensive evaluation involving over 30 open-weight and commercial LLMs. Our experiments provide a new perspective on Polish competencies in language models, moving past traditional natural language processing tasks and general knowledge assessment. 1 Introduction Recently, we have observed a dynamic development of large language models (LLMs). This led to profound changes in the approach to natural language processing. As a result, small models fine-tuned to perform specific tasks are gradually being replaced by large generative models, trained on massive text corpora and millions of instructions covering a wide variety of problems. From a business and end-user perspective, this simplifies the practical use of language models, since prompt engineering and few-shot learning are techniques that do not require specialized knowledge, as opposed to earlier approaches that involved re-training the models for each task. However, evaluating modern language models poses a significant challenge. Firstly, their ability to solve various types of tasks, and even generalize to tasks not seen during training, requires a multifaceted approach to evaluation. Secondly, these models generate responses in plain text, allowing a high degree of freedom in terms of response style and formatting. Assessing the correctness of a textual response is more challenging than evaluating a structured response such as a numerical value, class label, probability, or answer from a closed vocabulary. In recent years, various approaches to evaluating LLMs have been proposed. At the same time, there is a growing awareness of their drawbacks and limitations [ 3 , 21 , 19 ] . Therefore, it is crucial to use a combination of several benchmarks to verify various aspects of understanding and generation of natural language. One such aspect is multilinguality. Currently, an increasing number of models declare multilingual capabilities. Initially, these were mainly commercial models such as GPT-4 [ 25 ] , Claude [ 1 ] or Gemini [ 26 ] , but the trend has also extended to open-weights models. This publication focuses on the Polish language, which is often included among the supported languages. There are families of models that mention Polish directly (Qwen2 [ 35 ] , Command-R [ 12 ] , Aya [ 2 ] ) and those that declare broad support for multilinguality and are known to work well for Polish (Mistral [ 16 , 17 ] , Llama 3.1+ [ 9 ] ). It is also worth mentioning initiatives focusing solely on the Polish language such as SpeakLeash 1 1 1 https://speakleash.org/ or PLLuM [ 18 ] . Figure 1: Example showing various types of errors made by a multilingual model that was undertrained on Polish data. The user prompt is: How do you protect yourself from cold and flu? ( Jak chronić się przed przeziębieniem i grypą? ). In the model’s response, we can observe the following errors: a) morphological (green) - using the wrong form of a word; b) lexical (orange) - using a word that does not fit the context; c) word-formation (blue) - attempt to create a new word by imitating a word in another language, usually English; d) changing the language of the answer completely (yellow). The example includes snippets of the response generated by the Qwen2.5 7B model, using the interface provided by the model’s authors. For larger Qwen2.5 models the frequency of such errors decreases. In practice, the level of Polish proficiency among multilingual models leaves much to be desired. Models with a small number of parameters can struggle to generate coherent and grammatically correct text. As an example, Figure 1 shows the errors made by the Qwen2.5 model [ 35 ] when prompted to answer a simple question in Polish. For larger models, such problems are less frequent. However, the underrepresentation of Polish texts in the pre-training corpus can result in the model lacking Polish cultural context, which is essential for language understanding. The language includes references to historical events, folklore, literature, or pop culture can carry specific meanings in a particular culture but may not be understood or may have different associations in another culture [ 31 ] . As a result, models that on the surface may perform well in understanding and generating language will have difficulties correctly interpreting texts containing such references. 1.1 Contributions The goal of this publication is to examine both the linguistic and cultural competence of LLMs in Polish. To this end, we introduce a benchmark consisting of 600 manually prepared questions. The questions are divided into six categories representing different aspects of knowledge about Poland and the Polish language: history, geography, culture & tradition, art & entertainment, grammar, and vocabulary. Apart from thematic diversity, we also ensured diversity in the form of questions. In addition to questions that require a direct answer, the benchmark includes questions that involve combining facts, aggregating knowledge about different concepts, and sorting or filtering concepts according to specified characteristics. The evaluation of answers is deterministic and involves a rule-based system, inspired by benchmarks such as IFEval [ 37 ] or BBH [ 32 ] . For each question, we define a set of rules that must be met for the question to be considered as passed. For closed-ended questions, these rules are explicit and easy to define. For open-ended questions, the presence or order of certain words, phrases, or regular expressions is checked. As part of this study, we evaluated over 30 open-weights and commercial LLMs. We also launched a demo website 2 2 2 https://huggingface.co/spaces/sdadas/plcc featuring an up-to-date leaderboard, examples of questions from each category, and a description of our methodology. 1.2 Related work Most methods for evaluating large language models can be divided into four categories: \bullet Closed-ended questions and structured answers - Benchmarks of this type require the formulation of tasks for the LLM in such a way that the answer follows a format imposed by the authors so that it can be verified deterministically. The most common type of such tasks are single- or multiple-choice questions, in which the model is presented with answers from which it must select the correct ones. Among these types of tests, the popular ones include GPQA [ 27 ] and different variants of MMLU [ 14 , 34 ] . In addition, the group also includes tests that expect a structured answer, for example, datasets of math questions such as MATH [ 15 ] or GSM8K [ 7 ] . The category also incorporates many classic NLP tasks such

Read the complete text