LLMzSzŁ: a comprehensive LLM benchmark for Polish
Krzysztof Jassem, Michał Ciesiółka, Filip Graliński, Piotr Jabłoński, Jakub Pokrywka, Marek Kubis, Monika Jabłońska, Ryszard Staruch, Adam Mickiewicz University · 2025
Read LLMzSzŁ: a comprehensive LLM benchmark for Polish by Krzysztof Jassem, Michał Ciesiółka, Filip Graliński, Piotr Jabłoński, Jakub Pokrywka, Marek Kubis, Monika Jabłońska, Ryszard Staruch, Adam Mickiewicz University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02266.txt.
Introduction
Krzysztof Jassem and seven co-authors benchmark language models in Polish in twenty twenty-five. Their benchmark is the first at this scale for Polish. It is built from Polish national exams held by the Central Examination Board. Both academic and professional tests are included.
The set spans four exam types across one hundred fifty-four domains. It holds almost nineteen thousand closed-ended questions. The team tested open-source multilingual, English, and Polish models. They asked how well models transfer knowledge between languages. They also compared model accuracy with human pass rates. Multilingual models beat monolingual ones overall. But when model size matters, monolingual models can be the better buy. Readers get a school-desk test, as the benchmark's name jokes, for a mid-resource language.
Catalog record
| CREATOR | Krzysztof Jassem, Michał Ciesiółka, Filip Graliński, Piotr Jabłoński, Jakub Pokrywka, Marek Kubis, Monika Jabłońska, Ryszard Staruch, Adam Mickiewicz University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-02266
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLMzSzŁ: a comprehensive LLM benchmark for Polish Krzysztof Jassem, Michał Ciesiółka, Filip Graliński, Piotr Jabłoński, Jakub Pokrywka, Marek Kubis, Monika Jabłońska, Ryszard Staruch, Adam Mickiewicz University Abstract This article introduces the first comprehensive benchmark for the Polish language at this scale: LLMzSzŁ (LLMs Behind the School Desk). It is based on a coherent collection of Polish national exams, including both academic and professional tests extracted from the archives of the Polish Central Examination Board. It covers 4 types of exams, coming from 154 domains. Altogether, it consists of almost 19k closed-ended questions. We investigate the performance of open-source multilingual, English, and Polish LLMs to verify LLMs’ abilities to transfer knowledge between languages. Also, the correlation between LLMs and humans at model accuracy and exam pass rate levels is examined. We show that multilingual LLMs can obtain superior results over monolingual ones; however, monolingual models may be beneficial when model size matters. Our analysis highlight the potential of LLMs in assisting with exam validation, particularly in identifying anomalies or errors in examination tasks. 1 Introduction LLMs, as a form of Artificial Intelligence, have been invented to perform useful work in a specific environment defined by humans. Obviously, the question of how to evaluate LLMs has been raised since their inception Hendrycks et al. (2020) . Similarly, humans are expected to possess general and specialized knowledge to perform their work and, again, since the beginning of civilization, human capabilities have been assessed with general and professional exams. Humans, however, do not operate in an abstract vacuum. Their performance is always evaluated within a specific culture, its traditions, and legal system. Language leaves its stamp as well, not only because exams are always expressed in a particular language, but also because it is the lens through which reality is understood. One does not have to accept the Sapir-Whorf Hypothesis in its entirety to argue that expected answers are not always translatable between cultures and traditions. In this paper, we present a new Polish LLM benchmark called LLMzSzŁ ( LLM za Szkolną Ławą , LLMs Behind the School Desk ) representing a collection of Polish national exams. The characteristic feature of the dataset is the inclusion, in one coherent environment, of school and professional examinations. The incorporation of vocational exams makes it possible to advance hypotheses on LLMs’ specific competences for various professions that include: • practical application of knowledge in real professional situations, • real-life task solving, • acquaintance of industry regulations, safety standards and applicable laws, • competence in cutting-edge technology, • communication skills, • work organization. In a way, the benchmark represents what is expected from an ‘educated’ person to be able to navigate efficiently in the realities of modern Poland. The design choices we made while developing LLMzSzŁ distinguish it from other benchmarking efforts that rely on the collection of examination materials such as the MMLU proposed by Hendrycks et al. (2020) . First, instead of randomly searching the web for exams and collecting the results, we decided to rely on a single, reliable, authoritative source of tasks developed by educational experts. In our case, it is the collection of exams published by the Polish Central Examination Board (Pol. CKE, Centralna Komisja Egzaminacyjna) 1 1 1 https://cke.gov.pl/ which is an institution responsible for the development of testing materials for nationwide exams such as matriculation. Having a single source of tasks for the benchmark eliminates the problem of incorporating duplicate samples into the dataset. The credibility of the Polish Central Examination Board minimizes the risk of introducing questions of low merit or incorrect answers into the benchmark. Second, our dataset is stratified into the tiers that vary in the complexity of the tasks: middle school exams, which are usually taken at the age of 15, high school exams that require more elaborate language understanding and reasoning skills and are attempted at the age of 19, and vocational exams that along with reasoning skills also require domain-specific knowledge. Third, since the collected exam questions are originally written in Polish, the proposed benchmark can be used to verify if the models that were not deliberately trained for Polish are capable of solving the tasks by transferring the knowledge from the language of their origin. Finally, since we track the publication date for each exam, the proposed benchmark can be used to assess the performance of the model under study while minimizing the impact of data contamination on the evaluation results by restricting the set of test cases to those that were published after the release of the model. Furthermore, it enables the study of the correlation between the results achieved by the models and those of human examinees, and draw conclusions on a possible use of LLMs to check the difficulty of the future exams before their publication. Our main contributions are as follows: 1. Development of a new, multi-tier LLM benchmark that tracks publication time for all the collected samples. 2. Evaluation of a wide range of LLMs with a special focus on models developed for the Polish language. 3. Detailed analysis of the models’ performance with regard to the model size, language, release date, and model type (base vs. instruct). 4. Comparison of human and model performance in the presented tasks. 2 Related Work The capabilities of LLMs are verified by their performance against human-prepared benchmarks. In the pre-LLM era, linguistic benchmarks consisted of ”low-level tasks”, such as: grammatical correctness, sentiment analysis, paraphrase detection, semantic similarity, sentence entailment, natural language inference or coreference resolution Wang et al. (2018) . With the appearance of LLMs, emerging benchmarks have tended to focus on higher-level skills, such as reasoning Sawada et al. (2023) , problem-solving Zhang et al. (2023) or human-like conversation Liu et al. (2024) ; Bai et al. (2024) . A cost-effective way to design benchmarks to evaluate LLMs in combined aspects of knowledge, reasoning, and problem-solving is to build them on existing exams prepared for humans. Hendrycks et al. (2020) prepared the MMLU benchmark: 57 tests comprising 15,908 questions from humanities, social sciences, STEM and other subjects, collected from freely accessible sources. Later, MMLU was automatically translated (using, among others, DeepL and GPT-3.5) to produce the m_mmlu dataset for 35 languages 2 2 2 https://huggingface.co/datasets/alexandrainst/m_mmlu . A similar activity was conducted solely for the Spanish language by Plaza et al. (2024) with the use of Azure Translator. In the paper, a detailed analysis was carried out to discover the discrepancies between the performances of multilingual models for English and Spanish. Gema et al. (2024) identify numerous errors in the MMLU benchmark, which shows that there is still room for the development of new MMLU-like benchmarks, with a focus on their ground-truthfulness. The most recent trend in LLM benchmarking is to develop regional MMLU equivalents from national resources. Son et al. (2024) organized a dataset derived from Korean exams that covered 45 subjects from humanities and STEM. Yüksel et al. (2024) developed a smaller benchmark for Turkish (9 subjects), for which questions were prepared by high school experts. Li et al. (2024) created a dataset that covers four main categories: medicine, law, psychology, and education. Most of the questions were taken from university or college exams, but the legal tests were based