LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data

2025

Read LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24434.txt.

Introduction

Julian Valline and three co-authors introduce LuxIT, a large instruction tuning dataset for Luxembourgish, a low resource language. Instruction tuned models perform poorly where high quality training data is scarce. The team synthesized the dataset from native Luxembourgish texts using DeepSeek-R1, chosen for its strength in the language, then applied quality checks with a model as judge.

The process kept 227,507 high quality instruction and answer pairs. To test its value, the authors fine tuned fourteen smaller language models on LuxIT and evaluated them on Luxembourgish tasks. The dataset aims to bring capable instruction following to a language that commercial models often underserve.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24434

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data Abstract The effectiveness of instruction-tuned Large Language Models (LLMs) is often limited in low-resource linguistic settings due to a lack of high-quality training data. We introduce LuxIT, a novel, monolingual instruction tuning dataset for Luxembourgish developed to mitigate this challenge. We synthesize the dataset from a corpus of native Luxembourgish texts, utilizing DeepSeek-R1-0528, chosen for its shown proficiency in Luxembourgish. Following generation, we apply a quality assurance process, employing an LLM-as-a-judge approach. To investigate the practical utility of the dataset, we fine-tune several smaller-scale LLMs on LuxIT. Subsequent benchmarking against their base models on Luxembourgish language proficiency examinations, however, yields mixed results, with performance varying significantly across different models. LuxIT represents a critical contribution to Luxembourgish natural language processing and offers a replicable monolingual methodology, though our findings highlight the need for further research to optimize its application. Keywords: instruction-tuning, data augmentation, low-resource languages LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data Julian Valline, Cédric Lothritz, Jordi Cabot Luxembourg Institute of Science and Technology 5 Av. des Hauts-Fourneaux L-4362 Esch-sur-Alzette {julian.valline, cedric.lothritz, jordi.cabot}@list.lu Abstract content 1. Introduction In recent years, Large Language Models (LLMs) have demonstrated remarkable proficiency across a diverse range of natural language tasks ( Zhang et al. 2024 ; Brown et al. 2020 ; Touvron et al. 2023b ; Yin et al. 2023 ; Peng et al. 2023 ; Li et al. 2023a ) . Their rapid and widespread adoption as virtual assistants ( Shu et al. 2023 ) has been largely driven by their accessibility through intuitive chat-based interfaces ( Weber et al. 2024 ; Touvron et al. 2023b ) . The training of these assistants involves multiple steps, beginning with a pre-training stage where the model learns from a vast text corpus using a self-supervised objective like next-word prediction ( Weber et al. 2024 ; Touvron et al. 2023b ; Zhou et al. 2023 ) . The subsequent, critical step is Instruction tuning (IT) ( Wei et al. 2022 ) which refines the model’s ability to follow user instructions and engage in a conversational manner ( Weber et al. 2024 ; Zhang et al. 2024 ) . This process involves fine-tuning an LLM on an extensive dataset, which typically consists of instruction-answer pairs. By training on these pairs, the model learns to generate appropriate responses to user instructions, resulting in more anticipated and manageable outputs ( Zhang et al. 2024 ) . A notable benefit of this approach is that it can enhance the model’s ability to generalize and respond to previously unseen instructions ( Nayak et al. 2024 ; Zhou et al. 2023 ; Sanh et al. 2022 ) . A significant issue at present is the scarcity of open-source, non-English instruction tuning datasets, as the majority of existing IT datasets are predominantly in English, marginalizing other languages ( Weber et al. 2024 ) . This disparity leads to inferior performance for non-English languages and increases the cost associated with their training and deployment ( Weber et al. 2024 ) . This lack of resources is felt most acutely for low-resource languages, such as Luxembourgish, a West Germanic language spoken by about 600 000 people 1 1 1 https://cursus.edu/en/23040/luxembourgish-at-its-best , with its primary concentration of speakers found in Luxembourg. In this paper, we introduce LuxIT, a monolingual instruction tuning dataset in Luxembourgish, synthesized from articles of the news website RTL 2 2 2 https://www.rtl.lu/ and Wikipedia entries, both in Luxembourgish. Following the work of Lothritz and Cabot 2025 , we select DeepSeek-R1-0528, as the top-performing model for Luxembourgish generation and comprehension to synthetically produce 59 242 instruction-answer pairs. To validate the quality of our dataset, we assess the Luxembourgish capabilities of several LLMs before and after they have been fine-tuned on LuxIT. To our knowledge, no instruction tuning dataset for Luxembourgish currently exists that has been created exclusively from monolingual seed data for the purpose of fine-tuning state-of-the-art LLMs. Our primary contributions are twofold: • We introduce LuxIT 3 3 3 We provide a large subset sourced only from Wikipedia articles here , a synthetically generated instruction tuning dataset in Luxembourgish derived from monolingual seed data. • We examine the extent to which fine-tuning on LuxIT changes the Luxembourgish capabilities of 5 widely-used LLMs. 2. Related Work 2.1. Multilingual Instruction Tuning Datasets xP3 ( Muennighoff et al. 2023 ) is a multilingual, human-crafted instruction tuning dataset, where the data is taken from P3 and other multilingual datasets ( Zhang et al. 2024 ) . The dataset is constructed by combining data from various sources is combined into a unified format ( Zhang et al. 2024 ) . Muennighoff et al. 2023 further extend xP3 to xP3mt by applying machine translation ( Weber et al. 2024 ) . Multilingual datasets like Bactrian-X ( Li et al. 2023b ) emerged from machine-translated Alpaca instructions and matching, GPT-3.5-Turbo ( Brown et al. 2020 ) generated answers ( Weber et al. 2024 ) . LIMA ( Zhou et al. 2023 ) is a fine-tuned version of Llama ( Touvron et al. 2023a ) , trained on 1000 attentively selected instruction-answer pairs ( Zhang et al. 2024 ; Weber et al. 2024 ) . Weber et al. 2024 extend LIMA into Lima-X by translating the instructions from the original LIMA dataset into 4 languages. We refrain from using machine translation and multilingual seed-data and investigate the dataset generation in a low-resource setting from a strictly monolingual approach. While the model we use for data generation is pre-trained on a multilingual corpus, the data we feed to the model for synthesizing the instruction tuning dataset is only in Luxembourgish. 2.2. Synthesizing Instruction Tuning Data through Knowledge Distillation Taori et al. 2023 developed Alpaca, a modified Llama-7B model fine-tuned on 52k instruction-input-response triplets ( Zhang et al. 2024 ) , which are generated with text-davinci-003 ( Brown et al. 2020 ) . Alpaca performed on par with text-davinci-003 from OpenAI ( Zhang et al. 2024 ) . Peng et al. 2023 use GPT-4 ( OpenAI et al. 2024b ) to generate the output for the instruction-input pairs used in the original Alpaca dataset. Our work follows a similar dataset structure as Peng et al. 2023 , except that we use DeepSeek-R1-0528 to generate all new instructions and expected answers in Luxembourgish. 2.3. Luxembourgish Language Resources Recently, there have been various contributions to the creation of Luxembourgish language resources. Lothritz et al. 2022 developed a pre-training dataset in Luxembourgish through a data-augmentation technique where they partially translated text from German into Luxembourgish, resulting in LuxemBERT. Plum et al. 2024 presented LuxGen, a benchmark for evaluating data generation in Luxembourgish and LuxT5, an mT5-based ( Xue et al. 2021 ) text generation model in Luxembourgish, pre-trained on a German, French and Luxembourgish text corpus, where the latter is obtained through transfer learning from German and French. Most recently, Philippy et al. 2025b introduced a hand-crafted cross-lingual dataset for training Luxembourgish sentence embedding models, which led to the LuxEmbedder model. With the increasing ubiquity of LLMs, there is a need for a high-quality instruction tuning dataset in Luxembourgish to guide weaker models in improving their Luxembourgish capabilities. The cross-lingual dataset from Philippy

Read the complete text