Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora
Robert Litschko, Verena Blaschke, Diana Burkhardt, Barbara Plank, Diego Frassinelli · 2025
Read Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora by Robert Litschko, Verena Blaschke, Diana Burkhardt, Barbara Plank, Diego Frassinelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17855.txt.
Introduction
Robert Litschko and 4 co-authors build word lists that show how a dialect spells its words. Their 2025 study uses Bavarian, a German dialect with no fixed spelling rules. Large language models are rarely tested on such local speech, so their skill there is unclear.
A new frame, DiaLemma, builds the lists from plain text in one tongue only. Human checks gave a large set of German-Bavarian word pairs as ground truth. Nine top models were then tested on word sense and form. They did best on nouns and close word pairs, and worst at telling a true match from a bent form.
Catalog record
| CREATOR | Robert Litschko, Verena Blaschke, Diana Burkhardt, Barbara Plank, Diego Frassinelli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17855
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora Robert Litschko, Verena Blaschke, Diana Burkhardt, Barbara Plank, Diego Frassinelli Abstract Dialects exhibit a substantial degree of variation due to the lack of a standard orthography. At the same time, the ability of Large Language Models (LLMs) to process dialects remains largely understudied. To address this gap, we use Bavarian as a case study and investigate the lexical dialect understanding capability of LLMs by examining how well they recognize and translate dialectal terms across different parts-of-speech. To this end, we introduce DiaLemma , a novel annotation framework for creating dialect variation dictionaries from monolingual data only, and use it to compile a ground truth dataset consisting of 100K human-annotated German-Bavarian word pairs. We evaluate how well nine state-of-the-art LLMs can judge Bavarian terms as dialect translations, inflected variants, or unrelated forms of a given German lemma. Our results show that LLMs perform best on nouns and lexically similar word pairs, and struggle most in distinguishing between direct translations and inflected variants. Interestingly, providing additional context in the form of example usages improves the translation performance, but reduces their ability to recognize dialect variants. This study highlights the limitations of LLMs in dealing with orthographic dialect variation and emphasizes the need for future work on adapting LLMs to dialects. 1 1 1 https://github.com/mainlp/dialemma 1 Introduction Although most languages have standardized orthographies, this assumption does not hold for many languages and language varieties Millour and Fort 2020 . One prominent example are non-standard dialects, which have become increasingly popular in natural language processing (NLP) research Faisal et al. 2024 ; Joshi et al. 2025 . Dialects exhibit a large degree of spelling variation in written data, owing both to individual pronunciation differences and spelling preferences Just and Widmer 2025 . Such variation is commonly reflected in dialect NLP datasets ( Blaschke et al. 2023a ; Ramponi 2024 , cf.) . However, current NLP methods – which typically are trained on large amounts of standard-language data – are badly equipped to deal with such spelling variation, even if the word forms are similar to the standard. Issues range from subword tokenization Blaschke et al. 2023b ; Srivastava and Chiang 2025 , to evaluating generative tasks Aepli et al. 2023 , to retrieving information without being able to match word spellings Litschko et al. 2025 . 1. Judging Translation Pairs German Bavarian Translation ? zweisprachig zwaasprochig yes (“bilingual”) zwaspråchig yes zwoasprachign infl. dreisprochige (“trilingual”) no … … dazwischen dozwischn yes (“in between”) dawischn (“to catch”) no daktischen (“tactical”) no … … 2. Dialect-to-Standard Translation Bavarian German lemma zwaasprochig \rightarrow zweisprachig zwaspråchig \rightarrow zweisprachig dozwischn \rightarrow dazwischen … \rightarrow … Table 1: We annotate whether Bavarian dialect words with high string similarity to Standard German lemmas are (direct or inflected) translations (1). We test whether LLMs are able to do the same, and whether they can translate dialectal words into their standard-language counterparts (2). Therefore, it is important to build NLP systems that are either robust to variation or good at normalization. In order to gauge how good current or future systems are at this, we need evaluation datasets. However, prior works on collecting datasets of real-life dialectal spelling variation have resulted in relatively small datasets (Section 2 ). To allow large-scale analyses of the impact of dialectal spelling variation on NLP tasks, we conduct a case study on the Wikipedia edition in Bavarian, a dialect group related to German. This dialect wiki is comparatively large and has been used in prior work on dialect NLP Artemova and Plank 2023 ; Peng et al. 2024 ; Blaschke et al. 2024 ; Litschko et al. 2025 , and is included in pre-training datasets for, e.g., mBERT Devlin et al. 2019 . It covers the entire Bavarian-speaking area (regardless of subdialect), and explicitly encourages contributors to spell words as they find appropriate. 2 2 2 bar.wikipedia.org/wiki/Wikipedia:Wia_schreib_i_a_guads_Boarisch? “How do I write good Bavarian?”. Our approach only requires access to one monolingual corpus per language variety. Taking advantage of the fact that related standard and dialect varieties have a high number of cognates with similar word forms, 3 3 3 We acknowledge that there are other kinds of differences between standard and dialect varieties, e.g., lexical differences due to regional word choices. In this study, we focus on spelling differences between cognate words as the spelling variation is a specific challenge for NLP systems. we automatically extract German–Bavarian word pairs with high string similarities and annotate whether they are (direct or inflected) translations of each other (Table 1 ). We use our dataset to systematically analyze how well recent large language models (LLMs) can perform such annotations, and how good they are at translating the Bavarian terms into their German counterparts. Our study makes the following contributions: • We introduce a novel annotation framework for creating dialect variation dictionaries from monolingual data, without requiring parallel corpora (Section 3.1 ). • For 10k German words, we extract and manually annotate similarly spelled Bavarian words, resulting in 11k direct German–Bavarian translation pairs and 7k pairs with inflection differences (Section 3.2 ). • We evaluate nine state-of-the-art LLMs on two tasks: judging whether a given word pair is a translation (Section 5.1 ), and translating Bavarian words into their German counterparts (Section 5.2 ). We show that, overall, larger models perform better than their smaller versions, but they still struggle to distinguish between translations and inflected variants (Section 5.3 ). • We further show that (i) including additional context in the form of usage examples, on average, improves the model’s ability to translate and hurts the performance in judging translation candidates, (ii) using German prompts leads to worse results, and (iii) the Levenshtein distance between word pairs crucially impacts the model performance (Section 5.3 ). Figure 1: DiaLemma annotation framework. Step 1: Lemmatize the corpus of the majority language. Step 2: Extract the vocabularies from both the corpora of the majority language and dialect, respectively. Step 3: Filter the lemmas of the majority language out of the dialect vocabulary. Step 4: Find for each lemma the lexically most similar dialect terms using the Levenshtein distance. Step 5: Annotate if dialect terms are translations (‘yes’), dialectal inflected forms (‘inflected’), or neither (‘no’). 2 Related Work Inducing dialect and low-resource language dictionaries Artemova and Plank 2023 induce dialect dictionaries from parallel articles in German and dialectal wikis, extracting \sim 800 word pairs per language pair, and manually verifying them. Similarly, Haddow et al. 2013 extract German–Bavarian word lists from parallel and near-parallel sentences. Unlike these works, our approach does not rely on parallel data. Burghardt et al. 2016 and Burghardt 2023 crowdsourced German translations for 259 Bavarian words, while Millour and Fort 2019 collected spelling variants for 145 Alsatian words, resulting in about seven variants per word. Schmidt et al. 2020 manually translated a German word list into Swiss German dialects. Compared to these efforts, we do not require large-scale manual translations. Litschko et al. 2025 automatically induced translations for dialectal terms in seven German dialects and regional languages based on Wikipedia article titles and links, also collecting some spelling (and, rarely, lexical) variations. Unlike this approach, we are not limited to wiki