Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks
Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, Fabio Rinaldi, [0.5em] SUPSI, IDSIA, Switzerland · 2025
Read Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks by Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, Fabio Rinaldi, [0.5em] SUPSI, IDSIA, Switzerland (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20045.txt.
Introduction
Small dialect differences that humans shrug off can trip language models badly. Vani Kanjirangat and three co-authors trace part of that gap to tokenization. They measure two parities: how evenly tokenizers split text across languages, and how much information the tokens preserve. Those bias measures are then correlated with downstream results.
Three tasks are tested: dialect labeling, topic labeling, and extractive question answering. Modern decoder models face older encoder models across scripts and resource levels. Tokenization parity predicts best on tasks leaning on syntax and word form. The dialect gap has social and economic roots, but this work isolates a mechanical one. Fairer tokenizers are a concrete fix that model builders control directly.
Catalog record
| CREATOR | Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, Fabio Rinaldi, [0.5em] SUPSI, IDSIA, Switzerland |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20045
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, Fabio Rinaldi, [0.5em] SUPSI, IDSIA, Switzerland Abstract Dialectal data are characterized by linguistic variation that appears small to humans but has a significant impact on the performance of models. This dialect gap has been related to various factors (e.g., data size, economic and social factors) whose impact, however, turns out to be inconsistent. In this work, we investigate factors impacting the model performance more directly: we correlate Tokenization Parity (TP) and Information Parity (IP), as measures of representational biases in pre-trained multilingual models, with the downstream performance. We compare state-of-the-art decoder-only LLMs with encoder-based models across three tasks: dialect classification, topic classification, and extractive question answering, controlling for varying scripts (Latin vs. non-Latin) and resource availability (high vs. low). Our analysis reveals that TP is a better predictor of the performance on tasks reliant on syntactic and morphological cues (e.g., extractive QA), while IP better predicts performance in semantic tasks (e.g., topic classification). Complementary analyses, including tokenizer behavior, vocabulary coverage, and qualitative insights, reveal that the language support claims of LLMs often might mask deeper mismatches at the script or token level 1 1 1 Code at https://github.com/vanikanjirangat/Tokenizer_Fairness_Dialect . 1 Introduction Large Language Models (LLMs) pre-trained on massive text data in many languages have become the preferred solution for various Natural Language Processing (NLP) tasks. The use of this technology for processing dialects and regional varieties remains limited. Small variations in pronunciation and writing Zampieri et al. 2018 ; Scherrer et al. 2023 ; Habash et al. 2024 , which humans can easily ignore, lead to significant performance drops known as the dialect gap Kantharuban et al. 2023 . Including this variation is hard, although important for more human-like interactions with LLMs Amadeus et al. 2024 . It is especially important for a wide linguistic coverage, as many languages are not standardized or have multiple standards Samardžić and Ljubešić 2021 . In previous studies, dialect variances have been related to economic and social factors Kantharuban et al. 2023 , but the effects were inconsistent across different settings. Looking for more consistent factors directly related to how LLMs work, we turn to the representational biases in multilingual LLMs. We study two aspects where the biases can be quantified with recently proposed measures. First, the tokenization bias has been shown to impact not only the performance, but also the costs of deploying LLMs across languages Ahia et al. 2023 . Recently, this bias was quantified as Tokenization Parity (TP) Petrov et al. 2024 . Second, Information Parity (IP) Tsvetkov and Kipnis 2024 measures how well an LLM compresses or represents the same content across languages. In both cases, the measures show a difference between a given language and English as a reference language. To address the dialect gap, we correlate these measures to downstream performance on three dialect NLP tasks, each targeting a different level of representation: Dialect Identification (DI), which mostly relies on surface-level clues, Topic Classification (TC) as a primarily semantic task, and Extractive Question Answering (EQA) as a task that relies on both kinds of features. In all three cases, we work with multiple data sets representing different economic and cultural settings. This allows us to control for additional factors that are known to play an important role in creating biases. In particular, we control for the script (Latin vs. non-Latin) and resource level (high vs. low) van Esch et al. 2022 . On the model side, we control for the general type of pre-trained LLMs, distinguishing between encoder-only (BERT-based) and decoder-only (E.g., GPT) multilingual models. The Key Findings are: 1. Encoder-based models consistently outperform decoder-only LLMs 2 2 2 All models are referred to as LLMs and are distinguished as encoder-only or decoder-only where relevant across the evaluated dialectal tasks. 2. TP is more sensitive to the type of the script, while IP reflects biases influenced by both script and resource availability. Additionally, both metrics show model-dependent variation, highlighting how architectural and training differences contribute to representational disparities. 3. Information Parity (IP) shows more substantial alignment with tasks requiring semantic understanding and complex reasoning, while Tokenization Parity (TP) is more predictive for tasks that rely on morphological and syntactic features, especially span-based extractive tasks. These correlations are further modulated by language resource availability and script type. Figure 1: Tokenization Parity (TP) across languages resulting from the tokenizers used in encoder-type models. 2 Dialect Tasks, Data and Models Our selection of dialect NLP tasks, data and models was guided by the goal of covering as diverse settings as possible while keeping the computation feasible. 2.1 Tasks Dialect Identification (DI) This task consists of assigning a dialect or region label to each input sentence or utterance. This task comes in two versions: monolabel (each utterance can belong to only one dialect) and multilabel (some utterances can belong to multiple dialects), with the latter being more realistic but harder to perform and evaluate. We used the datasets from several VarDial shared tasks: Nuanced Arabic Dialect Identification (NADI-2023), Swiss German dialect identification (GDI), Indo-Aryan Language Identification (ILI), and multi-label DSL-ML datasets Abdul-Mageed et al. 2023 ; Samardzic et al. 2016 ; Zampieri et al. 2018 ; Chifu et al. 2024 . Topic Classification (TC) This task is similar to the monolabel DI task in that each snippet of text is assigned a single topic label. The difference is that predicting the label requires neutralizing surface-level differences between dialects. This task is included in the DialectBench benchmark Faisal et al. 2024 as the SIB-200 dataset Adelani et al. 2024 representing 200 languages. The topic classes include: {science/technology travel, politics, sports, health, entertainment, geography}. We conducted the fine-tuning experiments on 29 languages belonging to different scripts, along with the availability of language resources: eight Latin-high, nine Latin-low, five non-Latin-high, and seven non-Latin-low. Extractive Question Answering (EQA) This task combines in some way the features of the previous two, as it requires identifying the relevant spans (surface features) but relying on deeper semantic representation (understanding the question-answer relationships). We experimented on 24 dialectal variants - eleven Latin-high, two Latin-low, nine non-Latin-High, and two non-Latin-low, from the dataset SDQA Faisal et al. 2021 also provided via the DialectBench. The general statistics and further details of datasets are given in Appendix A . The class distributions for DI and TC tasks are shown in Figures 7 , 8 in Appendix A . 3 The Bias Metrics 3.1 Models & Tokenizers Encoder type We used the multilingual mBERT for the encoder variant in all the tasks. Specifically, in the case of the DI task, we performed additional comparisons between mBERT and language-specific variants such as MARBERT (Arabic), IndicBERT(Indic), German-BERT 3 3 3 We also did experiments with Swiss-BERT, which gave similar performance as German-BERT (Swiss-German), SpanBERTa (Spanish), and CamemBERT (French). We use the respective models from HuggingFace 4 4 4 https://huggingface.co/ . Decoder type Among the multilingual decoder-type models, we selected Phi-3.5-mini, Llama 3.2-3B, Mistral-7B, Falcon-7B, Gemma-7B, and SILMA-9B models. SILMA-9B represents an Arabic-specific LLM, while the other models discussed are English-centric or generalized multilingual LLMs, claiming support for a broad spectrum of languages. For the downstream task performance evaluation, we