Resource-Lean Lexicon Induction for German Dialects
2026
Read Resource-Lean Lexicon Induction for German Dialects (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23824.txt.
Introduction
Litschko, Plank, and Frassinelli show that simple statistical models beat large language models at inducing dictionaries for German dialects, a low-resource setting with wild spelling variation. Random forests trained on string-similarity features outperform a large open model, transfer across dialects, and stay lightweight. Using the induced dictionaries for query expansion lifts dialect retrieval scores by up to about 29 percent on one metric and 51 percent on another.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21626 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23824
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Resource-Lean Lexicon Induction for German Dialects Abstract Automatic induction of high-quality dictionaries is essential for building lexical resources, yet low-resource languages and dialects pose several challenges: limited access to annotators, high degree of spelling variations, and poor performance of large language models (LLMs). We empirically show that statistical models (random forests) trained on string similarity features are surprisingly effective for inducing German dialect lexicons. They outperform LLMs, enable cross-dialect transfer, and offer a lightweight data-driven alternative. We evaluate our models intrinsically on bilingual lexicon induction (BLI) and extrinsically on dialect information retrieval (IR). On BLI, random forests outperform Mistral-123b while being more resource-lean. On dialect IR with BM25, using our dialect dictionaries for query expansion yields relative improvements of up to 28.9% in nDCG@10 and 50.7% in Recall@100. Motivated by the resource scarcity in dialects, we further investigate the extent to which models transfer across different German dialects, and their performance under varying amounts of training data. Keywords: Dialect variation, dictionary, cross-lingual transfer, German dialects, low-resource languages Resource-Lean Lexicon Induction for German Dialects Robert Litschko 1,2 Barbara Plank 1,2 Diego Frassinelli 1 1 MaiNLP, Center for Information and Language Processing, LMU Munich, Germany 2 Munich Center for Machine Learning (MCML), Munich, Germany [email protected] Abstract content 1. Introduction The performance of current natural language processing (NLP) tools and large language models (LLMs) crucially depends on the language of use and how well it is represented in the (pre-)training corpus. High-resource languages like English and German benefit from robust machine translation (MT) and mature information retrieval (IR) systems. However, their performance has been shown to deteriorate when translating non-standard languages ( Gupta et al. 2025 ; Ziems et al. 2023 ) or retrieving relevant information where entities appear in regional spelling variations and word choices ( Litschko et al. 2025b ; Valentini et al. 2024 ; Chari et al. 2023 ) . This leads to a lexical dialect gap between majority languages and dialects. A major obstacle preventing the widespread adoption of NLP tools towards regional language varieties is the lack of resources and pretraining data, combined with the high variability due to the lack in standard orthography of many dialects. Multilingual language models cover only 1% of the world’s over 7,000 languages in existence ( Wang et al. 2022 ) and fail to accurately represent (i.e., tokenize) dialects at the input level ( Muñoz-Ortiz et al. 2025 ; Srivastava and Chiang 2025 ; Blaschke et al. 2023 ) . Dictionaries offer a resource-lean alternative to capture dialect variation. For example in the context of retrieval, dictionaries have been traditionally used to translate queries to the document language ( Ballesteros and Croft 1996 ; Adriani and van Rijsbergen 1999 ) and to induce cross-lingual embedding spaces ( Litschko et al. 2018 ) . In the context of translation, dictionaries have for example been used for data augmentation ( Kale et al. 2020 ; Waldendorf et al. 2022 ; Yin et al. 2024 ) , or to improve the ability of LLMs to translate texts into low-resource languages and rare words ( Ghazvininejad et al. 2023 ; Lu et al. 2024 ) . Compared to the (often unknown) language coverage of existing language models and translation systems, lexical resources exists for thousands of languages ( Wang et al. 2022 ) , with PanLex being the most prominent example ( Baldwin et al. 2010 ; Kamholz et al. 2014 ) . For dictionaries to be useful in practical applications they need to have a high lexical coverage (high recall), and high translation quality (high precision) to minimize erroneous translations. Meeting the coverage requirement is a particularly challenging task in the case of building dialect variation dictionaries, where words are mapped to multiple spelling variations. Manually curated dialect dictionaries developed by linguists achieve the highest quality standards, but are often unavailable in machine-readable form. We therefore focus on inducing bilingual dictionaries in a data-driven and lean way based on real-world dialect usage. Our method assumes to have access to Wikipedia for the target dialect, but is otherwise generalizable. To test the effectiveness of our approach, in this paper we specifically focus on five German dialects, each of which has its own Wikipedia: Alemmanic (als), Bavarian (bar), Ripuarian (ksh), Rhine Franconian (pfl), and Low German (nds). Prior work on inducing German dialect dictionaries relied heavily on human supervision ( Artemova and Plank 2023 ; Litschko et al. 2025a ; Litschko et al. 2025b ) . Artemova and Plank 2023 induce dictionaries based on word-alignments extracted from human-verified parallel sentences. This is not only costly and time-intensive due to the human supervision, but it also suffers from the same limitations as multilingual language models (i.e., most language varieties are not covered). Recent work tests LLMs for dictionary induction, but as shown in multiple studies ( Li et al. 2023 ; Merx et al. 2024 ; Litschko et al. 2025a , i.e.,) , building dictionaries with LLMs performs poorly in low-resource scenarios, especially for dialects. In contrast to these works, we show that statistical models trained on string similarity features are effective for building high-quality, high-coverage dictionaries at a much cheaper computational cost than LLMs, while achieving better overall quality. This opens up the support for more inclusive technology for minority languages. We further ablate the model performance with respect to the amount of training data used in the BLI task to determine how much data is truly needed to build high-quality resources. We make the following contributions: • We create dialect variation dictionaries for five German dialects (§ 2.4 ). • We validate the quality of our statistical model intrinsically on the task of BLI (§ 3.1 ), showing that it outperforms Mistral-123b. • We evaluate our induced dictionaries extrinsically on the task of cross-dialect information retrieval (§ 3.3 ), showing consistent improvements across all dialects. We make our code and resources available for future uptake. 1 1 1 https://github.com/mainlp/dialect-lexicon-induction 2. Methodology 2.1. Classifying Dialect Variation We frame the task of bilingual induction as a word-pair classification task. The starting point for this is a vocabulary in German l_{de} (majority language) and in a regional language variety l_{dial} (dialect). The goal is to match terms in l_{de} to one or more terms in l_{dial} . Since the quadratic combination of all possible matches is too large to explore in practice, we use the DiaLemma annotation framework ( Litschko et al. 2025a ) to obtain for each term up to k=10 lexical nearest neighbors, using Levenshtein distance ( Levenshtein 1966 ) . This allows us to pre-filter promising candidate dialect words prior to scoring word-pairs based on more sophisticated string similarity features. 2.2. String Similarity Features In this section, we describe the string similarity features used for our models, most of which have been used in Inkpen et al. 2005 in the context of classifying word pairs as cognates or false friends. The similarity measures are computed on pairs consisting of a German lemma x and a candidate dialect term y , and can be broadly grouped into set-based and sequence-based measures. Set-based measures first transform the dialect term y and German term x into sets of ngrams. The first measure has been proposed by Adamson and Boreham