Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English
Duke Nguyen, Aditya Joshi, Flora Salim · 2025
Read Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English by Duke Nguyen, Aditya Joshi, Flora Salim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12858.txt.
Introduction
Duke Nguyen, Aditya Joshi, and Flora Salim adapt language understanding models to dialects at test time in a 2025 study. Models are usually trained on standard American English, then judged on Indian, Singaporean, or Nigerian English, whose grammar and vocabulary shift noticeably. Labeled dialect data are scarce, so retraining for each variety is impractical.
Test-time adaptation adjusts the model using only the unlabeled inputs it is currently reading. That makes it attractive for dialect settings, where new text arrives with its variety baked in. The authors test how far such adaptation closes the dialect gap on understanding tasks, a fairness question for global English tools.
Catalog record
| CREATOR | Duke Nguyen, Aditya Joshi, Flora Salim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12858
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English Duke Nguyen, Aditya Joshi, Flora Salim Abstract Test-time adaptation (TTA) is an excellent method which helps generalize models across domains, tasks, and distributions without the use of labeled datasets. Thus, TTA is very useful in natural language processing (NLP) in the dialectal setting, since oftentimes, models are trained on Standard American English (SAE), evaluated on Indian English or Nigerian English, of which distribution differs significantly from the former. This is especially useful since dialectal datasets are scarce. In this paper, we explore one of the most famous TTA techniques, SHOT, in dialectal NLP. We finetune and evaluate SHOT on different combinations of dialectal GLUE. Our findings show that SHOT is a viable technique when labeled datasets are unavailable . We also theoretically propose the concept of dialectal gap and show that it has a positive correlation with the effectiveness of SHOT. We also find that in many cases, finetuning on SAE yields higher performance than finetuning on dialectal data. Our code is available at: https://github.com/dukenguyenxyz/dialect-adaptation . 1 Introduction Human languages exhibit natural variations with respect to four generally accepted dimensions: diaphasic (w.r.t. the medium of communication, e.g., texting versus oral communication), diastratic (w.r.t. social groups, e.g., gender), diachronic (w.r.t. time, e.g., 1960s versus 2010s English), and diatopic (w.r.t space, e.g., Indian versus Singaporean English) Zampieri et al. 2020 . These variations occur primarily in three domains: orthography, syntax, and vocabulary. Examples of diatopic varieties include: "I might could help you with that“ (Southern US, Australian, and New Zealand English), "Inside tent can not see leh! “ (Singaporean English) Joshi et al. 2025 . In this paper, we strictly study diatopic varieties, and we interchangeably refer to them as ‘dialects’ These natural linguistic variations pose significant challenges to NLP. Specifically, existing works have shown serious gaps with Large Language Models when processing Standard American English versus other forms of English. These gaps have been shown for bias in hate speech datasets Sap et al. 2019 , dependency parsing Blodgett et al. 2018 , automated speech recognition Koenecke et al. 2020 , syntactic processing Ziems et al. 2023 , exhibition of harmful stereotypes of speakers of AAE Hofmann et al. 2024 , exhibition of stereotyping, demeaning content, etc. towards speakers of non-SAE dialects Fleisig et al. 2024 . Another challenge is the significant number of English varieties. Kortmann et al. 2020 alone have documented 77 diatopic varieties of English, however DialectBench Faisal et al. 2024 , the most developed dialectal benchmark, only cover 19 varieties, many tasks of which are not equivalent. Building benchmarks and datasets also requires a significant amount of expense and annotation, which makes fine-tuning on labeled datasets more difficult to achieve. Recent trends in computer vision (CV), however, shows a promising new field, test-time adaptation (TTA), which can overcome these challenges. Test-time adaptation can potentially adapt an LLM trained on one dialect to another unseen dialect during inference time. In this paper, we study the use of one such TTA method, SHOT Liang et al. 2020 , in dialectal adaptation. We are the first to study dialectal adaptation using a TTA technique. We perform our evaluations across four dialects: SAE, Indian, Nigerian, and Singaporean across selected dialectal GLUE tasks. We compare SHOT performance against in-dialectal finetuning and cross-dialectal finetuning. Our results indicate that: (1) the wider the dialectal gap, the more effective SHOT is; (2) SHOT consistently improves the performance of models finetuned on SAE datasets and commonly outperforms models in-dialectal finetuned models; (3) cross-dialectal finetuning sometimes outcompetes in-dialectal finetuning. 2 Related Work There are limited English dialectal benchmark datasets. DialectBench Faisal et al. 2024 is large-scale NLP dialectal benchmark covering 10 text-level tasks and 281 varieties across languages. Mutli-VALUE Ziems et al. 2023 is a dialectal translation system which applies documented syntactic structure of dialects as studied by Kortmann et al. 2020 . Due to DialectBench Faisal et al. 2024 ’s limited tasks in English dialects, we have decided to use Multi-VALUE Ziems et al. 2023 for our evaluation. To overcome the dialectal gap, Dacon et al. 2022 apply adversarial learning to ensure consistence label prediction between the SAE and AAE inputs. Tan et al. 2020 propose a novel tokenization method which injects grammatical dialectical information as special symbols whilst reducing inflected words to their base form, improving the performance of AAE on SQUAD and MNLI. Held et al. 2023 use an architecture of dialectal adapters and task adapters via a contrastive loss and morphosyntactic loss. Xiao et al. 2023 use task-agnostic low-rank adapters to generalize an LLM to unseen dialects by leveraging expert linguistic knowledge via typological feature vectors. Srirag et al. 2024 similarly propose a task-agnostic low-rank adapter approach to the decoder model usecase. However, both rely on existing dialectal datasets. We first consider domain D of a joint distribution p(x,y) where x\in X,y\in Y are the input and output space, p_{S}(x,y) and p_{T}(x,y) are the distributions of the source and the target domain respectively. During test-time, we are given the source data D_{S}={(x_{1},y_{1}),...,(x_{n}s,y_{n}s))} and the unlabeled target data D_{T}={x_{1},...x_{n}t} Liang et al. 2023 . TTA aims to use self-supervised learning on D_{T} to improve the performance of the model originally trained on D_{S} without access to D_{S} . The most famous of TTA techniques is SHOT Liang et al. 2020 , which leverages the use of self-supervised pseudo-labeling in conjunction with information maximization in order to improve the performance of the model at test time. Previous works have shown promise in using TTA in the NLP domain Banerjee et al. 2021 ; Ye et al. 2022 ; Zhang et al. 2021 ; Wang et al. 2021 ; Ben-David et al. 2022 ; Laparra et al. 2021 . In addition, most have not adapted successful TTA techniques such as SHOT Liang et al. 2020 , except for Zhang et al. 2021 in domain adaptation. Task Train Eval Indian Nigerian Singaporean Average F TTA \Delta F TTA \Delta F TTA \Delta F TTA \Delta CoLA SAE 15.55 18.28 2.73 19.51 21.53 2.02 5.46 8.92 3.47 13.51 16.24 2.74 Indian 12.93 10.41 13.57 3.16 2.16 4.09 1.93 8.50 8.83 2.54 Nigerian 11.57 15.42 3.85 18.01 6.44 5.98 -0.47 12.01 10.70 1.69 Singaporean 13.03 15.76 2.73 13.73 12.47 -1.26 13.03 13.26 14.12 0.74 Average 13.27 16.49 3.10 15.42 15.86 1.31 6.77 6.33 1.64 11.82 12.89 2.02 SST-2 SAE 87.52 87.89 0.37 88.01 88.89 0.88 86.39 86.64 0.25 87.31 87.81 0.50 Indian 89.13 88.64 90.14 1.50 85.64 87.77 2.13 87.80 88.96 1.82 Nigerian 86.02 87.02 1.00 89.63 85.39 85.77 0.38 87.01 86.40 0.69 Singaporean 87.52 86.64 -0.88 87.52 88.26 0.74 88.38 87.81 87.45 -0.07 Average 87.55 87.18 0.16 88.45 89.10 1.04 86.45 86.73 0.92 87.48 87.67 0.71 RTE SAE 58.57 59.76 1.19 58.96 60.16 1.20 58.17 58.96 0.79 58.57 59.63 1.06 Indian 54.98 54.18 52.99 -1.19 53.78 54.98 1.20 54.31 53.99 Nigerian 58.96 55.78 -3.18 58.96 57.77 56.57 -1.20 58.56 56.18 -2.19 Singaporean 54.98 54.98 60.16 61.75 1.59 60.16 58.43 58.37 0.80 Average 56.87 56.84 -0.66 58.07 58.30 0.53 57.47 56.84 0.26 57.47 57.04 -0.08 Table 1: Matrix comparison of models fine-tuned and applied with TTA on different dialects then evaluated on other dialects. F stands for fine-tuning, \Delta=TTA-F