LID Models are Actually Accent Classifiers: Implications and Solutions for LID on Accented Speech
Niyati, Matthew · 2025
Read LID Models are Actually Accent Classifiers: Implications and Solutions for LID on Accented Speech by Niyati, Matthew (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00628.txt.
Introduction
Niyati Bafna and Matthew Wiesner present a 2025 speech technology paper with an accusation in its title: language identification models are really accent classifiers. Systems that name the language being spoken fail on accented speech, and the authors show why. The models often file a second-language speaker under their native language instead.
Experiments show the models shrug off shuffled short spans of speech, meaning they judge from brief sound patterns, the fingerprints of accent, rather than from word order or grammar. Two fixes follow: chopping input into chunks to force broader listening, and a sequence-aware design that needs no separate recognizer per language. Both cut accent-language confusion sharply while holding standard accuracy.
Catalog record
| CREATOR | Niyati, Matthew |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00628
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LID Models are Actually Accent Classifiers: Implications and Solutions for LID on Accented Speech Niyati, Matthew Abstract Prior research indicates that LID model performance significantly declines on accented speech; however, the specific causes, extent, and characterization of these errors remain under-explored. (i) We identify a common failure mode on accented speech whereby LID systems often misclassify L2 accented speech as the speaker’s native language or a related language. (ii) We present evidence suggesting that state-of-the-art models are invariant to permutations of short spans of speech, implying they classify on the basis of short phonotactic features indicative of accent rather than language. Our analysis reveals a simple method to enhance model robustness to accents through input chunking. (iii) We present an approach that integrates sequence-level information into our model without relying on monolingual ASR systems; this reduces accent-language confusion and significantly enhances performance on accented speech while maintaining comparable results on standard LID. 1 1 1 We release our code at https://github.com/niyatibafna/mitigating-accent-bias-in-lid/ . Bafna Wiesner nocounter] 1 CLSP and 2 HLTCOEJohns Hopkins UniversityUSA keywords language identification, accented speech, phonetic information † † email: [email protected], [email protected] 1 Introduction Spoken language identification (LID) is vital for speech processing pipelines, but often fails to generalize across diverse accents. Accent variation arises for many reasons: in national and global link languages such as English, Spanish, and Swahili, speakers color their pronunciation with the phonology of local substrate languages or their L1 languages [ 1 ] . For example, Indian English speakers may approximate the fricative /\theta/ in English with a dental /d/ , since most Indian language phone inventories do not contain the former. Other kinds of L1 accent variation may not be easily described by a currently spoken substrate or L1 language phonology, e.g., Latin American Spanish accents. While all speakers have an accent [ 2 ] , we focus specifically on how LID models handle accents influenced by local substrate or L1 substrate phonology -- L2-accents for short. 2 2 2 For bilingual speakers, or L1 speakers who share similar accent features as a group of L2 speakers we still call this accent an L2-accent. We aim to broadly characterize the mode of errors made by modern LID approaches on multiple L2-accents in 3 languages, to explain why some models fail to generalize to L2-accents, and finally, informed by our analyses, to improve LID robustness to L2-accented speech. To this end, we first study the error patterns of the ECAPA-TDNN LID model [ 3 , 4 ] on a variety of L2-accents. We find that accent-language confusion, i.e. the mis-recognition of L2-accented speech as the L1 substrate or a related language, is a significant contributor to this degradation. For example, Catalan-accented English is often identified as Catalan, Filipino-accented speech is identified as Tagalog and Cebuano, and so on. Accent-language confusion in an LID model indicates that the model functions as an accent ID model and simply relies on the strong correlation between accent and language to make language predictions. We hypothesize that this occurs when models use phoneme inventories or short phonotactic features characteristic of accent to make predictions, instead of lexical or syntactic features that may more generally characterize language. We probe LID models on adversarially constructed inputs designed to help characterize invariance to block permutations in order to help explain errors on L2-accented speech. Current commonly used LID models, such as the ECAPA-TDNN [ 4 ] , or pooled classification of self-supervised (SSL) representations of speech, borrow from techniques used in speaker ID. These models are not explicitly trained to capture sequence-level information, and involve pooling operations which inherently treat input sequences as exchangeable. For these reasons, they may be uniquely vulnerable to mis-classifying L2-accents. Formerly, phonotactic models were commonly used for LID [ 5 , 6 , 7 , 8 , 9 ] , and some recent work has looked at fusing these approaches to improve LID. Our analyses indicate that one reason for the success of these fusion approaches is that they are robust to L2-accents. Finally, we explicitly incorporate sequence-level views of the data into our models to improve accent-robustness. We explore two methods of extracting coarse sequence-level information from the input signal, and train transformer-based LID classifiers that take these as sequence inputs: a) phonetic transcripts ( phoneseqs ), and b) clustered SSL representations ( duseqs ). The latter is inspired by [ 10 , 11 , 12 ] , who show that discrete SSL units largely encode phonemic information. We show these models display very little accent-language confusion, displaying largely accent-agnostic error patterns for English. This lends support to the claim that sequence-level features contribute to accent-robustness for LID. However, these models show poor overall LID performance as compared to acoustics-based SOTA model. This indicates that acoustic and phonetic representations provide complementary information and can therefore be beneficially combined. We explore combinations of phonetics-based and acoustics-based views of the data, including fusion of model output distributions ( ET+phoneseqs ), as well as using frozen ECAPA-TDNN representations with trainable phonetics-based modules ( ET+phoneseqs-train , ET+duseqs-train , and ET+duseqsembed-train ). Our best-performing model, ET+phoneseqs-train , shows large gains on L2-accents (up to +34 LID points for English L2 accents), while suffering minimally on LID for mainstream accents. Our work sheds light on a prevalent mode of and reasons for the failure of SOTA LID systems on accented speech, shows that sequence-view-based LID models are less susceptible to this problem, and demonstrates the benefits of incorporating local and global views of the data on accented speech. 2 Related Work The degradation of commonly used LID and ASR systems on accented speech is well documented [ 13 , 14 , 15 , 16 , 17 ] , although largely only for English dialects and accents. We extend this analysis to a broader set of accents and languages than previously studied for LID systems. Some previous work has investigated using ASR outputs to aid LID systems on accented speech: [ 18 ] use ASR hypotheses for multiple monolingual ASR systems as input to an LID system, showing improvements on accented and code-switched speech. Other studies have also explored improving LID by using phonetic information; [ 19 ] show that fine-tuning wav2vec2 on articulatory feature detection improves its performance on English/Mandarin LID. [ 20 ] proposed PHO-LID, which uses an additional SSL-based contrastive phoneme segmentation objective to inject phonotactic information into the model and reduces confusions between nearby languages. Our modeling approach, while similar, is somewhat simpler and we primarily use it to analyze how phonotactic information improves performance on L2-accents. Our work is similar to [ 21 ] , which uses a naïve Bayes char-gram model monolingual-ASR transcripts to inform LID predictions using fusion with the SOTA model predictions. To our knowledge, ours is the first work to characterize accent-language confusion as a major cause of failure for LID on accented speech, and link it with the block permutation invariance of LID models. Our model uses a language-agnostic phonetic transcriber instead of relying on ASR systems, and is trained with access to SOTA model representations, which outperforms a shallow fusion-based approach. 3 Datasets See Table 1 for a summary of the datasets we used. General domain training – The VoxLingua-107 (VL-107)