Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic
2026
Read Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16406.txt.
Introduction
Ingimundarson and co-authors audit language model benchmarks for Icelandic. Many tests for smaller languages use synthetic or machine-translated items. Such items often carry severe flaws that skew scores. The team ran a quantitative error analysis on current Icelandic benchmarks. Human-made tests clearly differed from synthetic or translated ones. They warn that unverified machine translation caps test quality at translation quality. Better evaluation methods are needed for low-resource languages.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46440 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16406
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic Abstract This paper evaluates current Large Language Model (LLM) benchmarking for Icelandic, identifies problems, and calls for improved evaluation methods in low/medium-resource languages in particular. We show that benchmarks that include synthetic or machine-translated data that have not been verified in any way, commonly contain severely flawed test examples that are likely to skew the results and undermine the tests’ validity. We warn against the use of such methods without verification in low/medium-resource settings as the translation quality can, at best, only be as good as MT quality for a given language at any given time. Indeed, the results of our quantitative error analysis on existing benchmarks for Icelandic show clear differences between human-authored/-translated benchmarks vs. synthetic or machine-translated benchmarks. Keywords: benchmarking, machine translation, low-resource languages, Icelandic Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic Finnur Ágúst Ingimundarson 1 , Steinunn Rut Friðriksdóttir 2 , Bjarki Ármannsson 2,3 , Iris Edda Nowenstein 2 , Steinþór Steingrímsson 3 1 University of Zürich, 2 University of Iceland, 3 The Árni Magnússon Institute for Icelandic Studies [email protected], {srf, irisen}@hi.is, {bjarki.armannsson, steinthor.steingrimsson}@arnastofnun.is Abstract content 1. Introduction Benchmarking has been an integral part of development in NLP as a way of assessing performance on various tasks and traditionally on tasks with a single-label output, such as POS-tagging, NER, sentiment analysis, etc. With the advent of LLMs, that are not only capable of performing all of these traditional tasks and yet more ( 7 ) but perhaps even of creating such data, the need for robust benchmarks is even greater than before. In their wake also follows a demand for creating new benchmarks that are better adapted to the capabilities of LLMs, i.e. beyond evaluating single-label output, and importantly, beyond tasks which purely target the scope of expertise of NLP researchers ( 6 ) . This means that benchmark set creators increasingly find themselves creating benchmarks for natural language production and understanding in languages they do not speak and within subjects of which they are not experts. For example, there is a clear difference between constructing a NER evaluation metric in a language the researcher knows and generating a machine-translated benchmarking set for the evaluation of an LLMs’ capabilities in medical text summarization ( 1 ) . Furthermore, the field – and therefore benchmarking – has largely been dominated by English (and a few other high-resource languages ( 40 ; 43 ) ), but given the multilingual capabilities of LLMs, it is important to put them to the test in a valid manner. For one thing, it can be informative to see how different scripts or more morphologically rich languages compare. This is of particular importance for low-resource languages, as they run the risk of being submerged in the digital ocean of English data. For low- and medium-resource languages, it is often hard to keep up with the pace and establish what already exists for high-resource languages. Benchmarking is no exception. As benchmark production with human input can be time-consuming and costly, one common approach in such settings is to machine-translate existing benchmarks for other languages or use LLMs to create synthetic benchmarks, for instance by generating question-answer pairs from pre-existing data such as Wikipedia. But what are the drawbacks to this approach and do they affect low-resource languages disproportionally? In this paper we look at the current state of benchmarking for Icelandic as a case study of a low/medium-resource language, discuss the effects and results including machine-translated and synthetic data, and present the results of a quantitative error analysis on existing benchmarks for Icelandic. They show clear differences between human-authored or human-translated benchmarks vs. synthetic or machine-translated benchmarks, especially where the output has not been verified and/or corrected. 1 1 1 All code and data available at https://github.com/finaing/who_benchmarks_the_benchmarks 2. Related Work Discussion on the pitfalls and problems of language model benchmarking, and the question of how to best compare the capabilities of different models, has been ongoing in the NLP community for years. At least as early as 6 , it had become apparent that the best-performing NLU models of the day, though far from perfect in real-world applications, had almost completely saturated existing benchmarks, begging the question of how best to measure actual progress. Four years later, this is still an open and important question as evidenced by 13 which identifies ‘key shortcomings’ in benchmarking practices over the last decade, including ‘issues in the design and application’, ‘misaligned incentives’, ‘construct validity issues,’ and other shortcomings directly related to the points made in this paper. As previously discussed, it has become commonplace practice in recent years to automate the construction of benchmarks and evaluation sets either fully or partially. Here, statistical power (which is certainly important, see e.g. 8 ) is seemingly prioritized over precision and overview, despite the fact that machine learning models are frequently ‘right for the wrong reasons’ ( 26 ) , capable of achieving high scores through learning unintended shallow heuristics. Indeed, there is considerable evidence that smaller, carefully curated test sets are often equally good or better indicators of performance than larger, synthetic sets (see e.g. 17 ; 33 ; 46 ). This is in line with the importance of validity in any kind of testing. Present already in e.g. 12 , validity is a key concept in psychometrics and addresses whether a test or evaluation instruments truly measures what it intends to measure (the construct). In the context of LLM testing and particularly in clinical applications, calls have been made to prioritize construct validity and employ a Benchmark-Validation-First Approach to (medical) LLM evaluation ( 1 ) . To paraphrase, benchmarks need to be right for the right reasons. Even automatically constructing test items using a structured template, now the norm for grammatical evaluation sets ( 44 ; 19 ; 37 ; 23 ) , can easily result in semantically bizarre examples and items solvable through superficial heuristics rather than genuine linguistic competence ( 41 ) . Similarly, the widespread practice of evaluating systems for different languages by automatically translating existing benchmarks (usually from English) has repeatedly been shown to result in translation artifacts which affect accuracy and omit language- and culture-specific context ( 10 ; 28 ; 9 ; 25 ) . For instance, 39 examine 17 Turkish datasets (a “mid-resource” language), many of which are translations/adaptations from English. They identify major shortcomings related to coherence, fluency, cultural alignment, and increased risk of bias and error. Similarly, 48 note major shortcomings in culturally specific machine translations where exact translation pairs might not exist. Furthermore, 32 demonstrate shortcomings of template-based translations in multilingual benchmarks, showing that sentence-level translations yield more reliable results. As noted by 40 , multilingual benchmarks remain limited in scope and disproportionately represent high-resource languages, neglecting the cultural and linguistic diversity of lower-resource languages. Moreover, benchmarks are often absorbed into training corpora, resulting in contamination and inflated evaluation scores 47 . When evaluation datasets rely on translations of English benchmarks without cultural adaptation, models may optimize for unnatural constructions rather than authentic usage. This issue is amplified in small-language settings, where limited corpora increase the influence of benchmarks on model optimization.