DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching

Jessica Ojo, Zina Kamel, David Ifeoluwa Adelani · 2025

Read DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching by Jessica Ojo, Zina Kamel, David Ifeoluwa Adelani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17768.txt.

Introduction

Jessica Ojo, Zina Kamel, and David Ifeoluwa Adelani test how well tools can tell which tongue a text is in. Their 2025 paper calls that task language identification. It is a first step for most tools that work across tongues, yet tests tend to use clean, single-tongue text.

Their bench, DIVERS-Bench, checks speech text, web text, social posts, kids' tales, and mixed text. Top tools did well on clean sets but fell hard on noisy, casual text. A new mixed set with ten tongue pairs also tripped tools up. The work shows a need for tools that cope with real, messy use.

Catalog record

CREATORJessica Ojo, Zina Kamel, David Ifeoluwa Adelani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-17768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching Jessica Ojo, Zina Kamel, David Ifeoluwa Adelani Abstract Language Identification (LID) is a core task in multilingual NLP, yet current systems often overfit to clean, monolingual data. This work introduces Divers-Bench a comprehensive evaluation of state-of-the-art LID models across diverse domains - including speech transcripts, web text, social media texts, children’s stories, and code-switched text. Our findings reveal that while models achieve high accuracy on curated datasets, performance degrades sharply on noisy and informal inputs. We also introduce Divers-CS , a diverse code-switching benchmark dataset spanning 10 language pairs, we show that existing models struggle to detect multiple languages within the same sentence. These results highlights the need for more robust and inclusive LID systems in real-world settings. 1 1 1 https://github.com/McGill-NLP/DIVERS-Bench 1 Introduction Language identification (LID) which is the task of determining the language of a given text, is foundational in multilingual NLP, enabling applications such as translation, speech recognition, and content moderation ( Burchell et al. 2023 ) . While modern LID systems report high accuracy on benchmark datasets and cover hundreds of languages ( Goswami et al. 2024 ; Jauhiainen et al. 2019 ; Kargaran et al. 2023 ) , their real-world performance remains limited. State-of-the-art LID models often evaluate on clean, high-resource, single-label data and struggle with noisy, informal, or code-switched inputs. Real-world text frequently includes slang, misspellings, and dialectal variation ( Caswell et al. 2020 ) , yet evaluations remain dominated by curated datasets such as WiLI ( Thoma 2018 ) , UDHR ( Kargaran et al. 2023 ) , and FLORES ( Goyal et al. 2022 ) , which fail to capture this complexity Prioleau & Aryal 2023 . This study presents a comprehensive evaluation of LID robustness across three axes: domain diversity, language resource imbalance, and code-switched inputs. We focus on two questions: (1) How robust are current LID models to noisy and informal text? (2) How well do they handle code-switching across diverse language pairs? We evaluate eight LID models on datasets spanning speech transcripts, social media, children stories, and translation corpora. We also introduce a diverse code-switching (CS) benchmark covering 10 language pairs across high- and low-resource settings, the most comprehensive code-switching dataset to date. Our evaluation framework provides a more inclusive view of LID performance across real-world conditions. 2 Related Works Language identification (LID) is a critical preprocessing step in multilingual NLP, supporting tasks such as translation, transcription, and parsing ( Panich et al. 2015 ) . Early LID systems relied on character n-gram features with classifiers like Naive Bayes and SVMs ( Baldwin & Lui 2010 ) . With the rise of deep learning, scalable neural models have emerged. FastText-LID ( Joulin et al. 2017 ) , a shallow n-gram-based model trained on Wikipedia, remains widely used. Its architecture has been extended in systems like GlotLID ( Kargaran et al. 2023 ) , covering over 1,600 languages. Other notable models include CLD3 (LSTM-based), AfroLID ( Adebara et al. 2022 ) for African languages, MaskLID ( Kargaran et al. 2024 ) for code-switching, and OpenLID ( Burchell et al. 2023 ) , which focuses on curated multilingual sources. Most LID models report strong performance on benchmarks such as WiLI-2018 ( Thoma 2018 ) , Tatoeba ( Tiedemann 2020 ) , LangID-33 ( Lui & Baldwin 2012 ) , and FLORES Goyal et al. 2022 . However, these datasets are typically clean, formal, and monolingual, and thus fail to reflect real-world variation, including noisy, informal, short, or mixed-language inputs ( Caswell et al. 2020 ; Martins & Silva 2005 ; Doğruöz et al. 2021 ) . Studies such as Zubiaga et al. 2016 ; Carter et al. 2013 show that even state-of-the-art models degrade substantially on social media and microblogs, which often feature slang, emojis, dialects, and non-standard grammar. In addition, real-world language use frequently involves the alternation between two or more languages within a single sentence or utterance referred to as CS which poses a significant challenge for existing LID systems ( Burchell et al. 2024 ; Winata et al. 2023 ) . While corpora like TongueSwitcher ( Sterner & Teufel 2023 ) and CanVEC ( Nguyen & Bryant 2020 ) address CS, they typically focus on narrow language pairs or domains, limiting generalizability. OpenLID has extended evaluation to a handful of CS scenarios, but broader coverage remains scarce. Our research addresses this gap by building upon and complementing existing work. We perform an extensive evaluation of SoTA LID models on a diverse collection of formal and informal datasets, including speech transcripts, web-crawled text, social media, children’s stories, and professional translations. We also build on these efforts by introducing Divers-CS a diverse CS benchmark covering 13 language pairs - spanning both high- and low-resource languages to provide a more inclusive evaluation of LID robustness in mixed-language settings. 3 Evaluation Strategy 3.1 LID Models We evaluate eight LID models spanning a broad range of techniques, language coverage, and training regimes, as summarized in Table 2 . These include traditional models such as Google’s Compact Language Detector 3 (CLD3), LangDetect, and LangID.py ( Lui & Baldwin 2012 ) , which rely on statistical, frequency-based, or rule-based methods. Data-driven approaches include FastText LID ( Joulin et al. 2016 ) and Franc, while recent large-scale systems such as GlotLID ( Kargaran et al. 2023 ) , OpenLID ( Burchell et al. 2024 ) , and ConLID ( Foroutan et al. 2025b ) leverage multilingual corpora and deep learning. ConLID, in particular, employs supervised contrastive learning to improve robustness on out-of-domain and low-resource languages. Appendix B describes models evaluated. 3.2 Divers-Bench : Datasets and Domains We introduce Divers-Bench a new benchmark covering several domains and real-world CS corpora. To evaluate the robustness of LID models across diverse linguistic and real-world conditions, we selected five datasets spanning clean, noisy, informal, and speech-based text sources. These datasets vary in language coverage, domain, and text quality. Table 2 details key dataset properties. 3.2.1 Divers-Mono : Monolingual data FLORES-200 ( Goyal et al. 2022 ) provides professionally translated sentences across 200 languages, for a clean and standardized benchmark, particularly for low-resource languages. SMOL ( Caswell et al. 2025 ) is a multilingual benchmark dataset consisting of professionally translated sentence- and document-level data for 115 LRLs. SMOL highlights linguistic diversity by incorporating related, endangered, and underrepresented languages, and includes sources from diverse domains. TweetLID ( Zubiaga et al. 2016 ) is composed of annotated Twitter data, rich in abbreviations, code-switching, and informal expressions, and messy texts for evaluating robustness. Bloom stories ( Leong et al. 2022 ) is a dataset of publicly accessible children’s stories translated into over 100 languages, providing high-quality but simpler text that reflects natural and structured storytelling patterns that differ from other domains. Common Voice ( Ardila et al. 2020 ) is a multilingual read-speech dataset that collects texts from language communities and records them being read aloud. For our evaluation, we focus only on the sentences. Model Technique #Langs CLD3 n-gram Neural Net 100+ FastText Subword embeddings 176 Franc Trigram frequency 414 LangDetect Statistical + rule-based 24 LangID Naïve Bayes 97 GlotLID FastText 2000+ OpenLID FastText 200+ ConLID FastText + SCL 2000+ Table 1: Evaluated LID

Read the complete text