ltzGLUE : Luxembourgish General Language Understanding Evaluation

Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cédric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke · 2026

Read ltzGLUE : Luxembourgish General Language Understanding Evaluation by Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cédric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17976.txt.

Introduction

Plum and ten colleagues present ltzGLUE, the first natural language understanding benchmark for Luxembourgish, a national language often overlooked although tasks exist for many European tongues. They build new tasks and reuse old ones, covering binary and multi-class settings, including named entity recognition and topic and intent sorting. The paper also evaluates various pretrained encoder models on Luxembourgish, giving an overview of current model skills in the language.

Catalog record

CREATORAlistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cédric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55251
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17976

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ltzGLUE : Luxembourgish General Language Understanding Evaluation Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cédric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke Abstract This paper presents ltzGLUE , the first Natural Language Understanding (NLU) benchmark for Luxembourgish (LTZ) based on the popular GLUE benchmark for English. Although NLU tasks are available for many European languages nowadays, LTZ is one of the official national languages that is often overlooked. We construct new tasks and reuse existing ones to introduce the first official NLU benchmark and accompanying evaluation of encoder models for the language. Our tasks include common natural language processing tasks in binary and multi-class classification settings, including named entity recognition, topic classification, and intent classification. We evaluate various pre-trained language models for LTZ to present an overview of the current capabilities of these models on the LTZ language. 1 Introduction Language models now support Natural Language Processing (NLP) tasks in more languages than ever before 27 . Advances since the introduction of the Transformer architecture 44 ; 8 ; 42 have led to substantial performance gains, enabling Large Language Models (LLMs) to achieve state-of-the-art results across a wide range of tasks. As a result, both closed and open-weight LLMs have become the models of choice in NLP and related fields. Owing to their architecture and exposure to large-scale multilingual pre-training data, these models often demonstrate strong performance across many languages. Moreover, their ability to be fine-tuned for a wide variety of downstream tasks enhances their multilingual capabilities. The perceived support for a wide range of languages has created an unprecedented need for language-specific evaluation of language models. As access to LLMs becomes increasingly widespread, so too does the belief that these models perform well across all languages, an assumption that does not always hold in practice 55 . In the interest of transparency and responsible deployment, it is therefore essential to systematically evaluate the Natural Language Understanding (NLU) capabilities of language models 14 . Small and under-researched languages are particularly difficult to evaluate, as is the case with Luxembourgish (LTZ), the national language of Luxembourg, with around 400k speakers. In English, multiple benchmarks for NLU exist, including GLUE 46 and SuperGLUE 45 , and more for other large and small languages alike 28 ; 2 ; 13 ; 38 . However, this is not the case for LTZ, which only has a handful of NLU tasks available 25 ; 29 ; 30 . As most of these are in the news domain, and the majority of the down-stream tasks comprise less than a thousand instances, model evaluation is not always dependable. Additional factors, such as the ongoing standardisation of the language 11 , vast amounts of variation 26 , and decentralised resources, make it extremely challenging to evaluate LTZ language understanding in language models. To address these gaps, we introduce ltzGLUE , a general language understanding benchmark for LTZ that includes new and existing NLU tasks. Moreover, we evaluate various pre-trained language models for LTZ to ascertain the current state of the art for the language. Our contributions are: (1) ltzGLUE : the first unified GLUE benchmark for LTZ, with 8 tasks. 1 1 1 https://github.com/plumaj/ltzGLUE (2) ltz-E1 (mini/base): 2 new encoder language models for LTZ, which achieve competitive performance when fine-tuned on ltzGLUE . 2 2 2 https://huggingface.co/instilux (3) A systematic evaluation of new and existing models for LTZ. 2 Related Work Work on language understanding has progressed along two largely separate lines: large-scale benchmarking for high-resource languages and emerging efforts to build resources for smaller ones. The first has produced influential frameworks which have shaped evaluation practices for pre-trained models across domains. The second has focused on adapting NLP methods to under-researched languages, where data scarcity and linguistic variation remain major challenges. The General Language Understanding Evaluation (GLUE) benchmark 46 became a cornerstone of NLU research by consolidating diverse tasks such as sentiment analysis 41 , textual entailment 50 , and paraphrase detection 9 into a unified evaluation framework. It established a shared reference point for pre-trained models like BERT 8 , RoBERTa 23 , and XLNet 53 , and allowed for systematic comparison across architectures and training regimes. SuperGLUE 45 addressed some shortcomings by introducing more challenging tasks such as COPA 36 , WSC 20 , and MultiRC 18 , shifting focus toward commonsense inference and multi-sentence reasoning. While both benchmarks remained English-only, the methodological influence extended widely, shaping later evaluation design in terms of robustness, transparency, and reproducibility. Multilingual benchmarks in the GLUE style have also been developed, including XGLUE 21 , XTREME 15 , and XTREME-R 37 , as well as language-specific adaptations such as KLUE 28 , RussianSuperGLUE 38 , bgGLUE 13 and sinhalaGlue 34 . These efforts highlighted the limits of cross-lingual transfer, reinforcing the need for careful, language-specific evaluation beyond high-resource settings. 2.1 Luxembourgish NLP LTZ, the focus of this benchmark, is regarded as under-researched, and research is ongoing. 17 classify Luxembourgish as one of the “scraping-by” languages: although some unlabeled data exists, meaningful progress will require coordinated efforts to raise awareness and collect labeled datasets, as such resources are currently almost nonexistent. Nevertheless, the first computational tools and corpora were introduced by 1 , followed by orthographic studies such as contextual n -deletion in transcribed speech 40 . 19 later provided one of the earliest annotated datasets for mixed-language processing. These efforts, though limited, established the foundations for subsequent large-scale data creation and model development. Since then, the range of tasks has expanded considerably. Work on sentiment analysis 39 ; 10 , orthographic correction 33 , and syntactic annotation 31 has broadened the empirical basis for LTZ NLP. Additional datasets have targeted topic classification 29 , comment moderation 35 , and orthographic normalisation 26 , alongside the generative benchmark set LuxGen 32 . A manually annotated classification resource was introduced by 25 , which covers a variety of classification tasks. Model development has explored different transfer and training strategies. LuxGPT 3 applied cross-lingual transfer from German, LuxemBERT 25 used data augmentation with generated samples, and LuxT5 32 extended multilingual pre-training with a balanced language representation. Yet progress remains uneven across tasks, and existing resources vary widely in size, domain, and annotation quality. No unified benchmark currently exists to evaluate LTZ language understanding consistently, a gap we aim to fill. 3 Tasks In this section, we introduce the eight tasks for ltzGLUE . The set spans binary and multi-class sentence and token-level classification tasks. Together, these tasks cover a broad spectrum of linguistic and semantic phenomena and provide the first unified benchmark for evaluating LTZ NLP models. See Table 7 in the Appendix for task examples. Unless stated otherwise, the textual data used across most tasks stems from two main sources: (i) RTL 3 3 3 https://rtl.lu is the main news broadcaster in Luxembourg, and the only one that is completely in LTZ. RTL provides news articles from the time span 2008 until 2024. (ii) Wikipedia has a growing set of articles in LTZ, around 66k at the time of writing. 3.1 Headline Acceptability We formulate headline acceptability ( HA ) as a binary classification task where the model must decide whether a given headline

Read the complete text