CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera-Gómez, Sara Hincapie-Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Otiende, Tack H · 2026

Read CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data by Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera-Gómez, Sara Hincapie-Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Otiende, Tack H (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18026.txt.

Introduction

Pedro Ortiz Suarez and co-authors re-test how well machines can tell one language from another. Their 2026 preprint presents CommonLID, a benchmark for language identification. Telling languages apart is a first step in building multilingual corpora. Yet models still do poorly for many languages, above all on noisy web text. Web text is exactly what trains today's multilingual models, so errors flow downstream.

CommonLID is community-driven and labeled by people. It covers 109 languages in the web domain. A shared, honest benchmark shows where models fail and for whom. It also gives builders a target to improve against. Better language ID means cleaner training data for low-resource languages.

Catalog record

CREATORPedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera-Gómez, Sara Hincapie-Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Otiende, Tack H
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18026

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera-Gómez, Sara Hincapie-Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Otiende, Tack Hwa Wong, Jakhongir Saydaliev, Melika Nobakhtian, Muhammad Ravi Shulthan Habibi, Chalamalasetti Kranti, Carol Muchemi, Khang Nguyen, Faisal Muhammad Adam, Luis Frentzen Salim, Reem Alqifari, Cynthia Amol, Joseph Marvin Imperial, Ilker Kesen, Ahmad Mustafid, Pavel Stepachev, Leshem Choshen, David Anugraha, Hamada Nayel, Seid Muhie Yimam, Vallerie Alexandra Putra, My Chiffon Nguyen, Azmine Toushik Wasi, Gouthami Vadithya, Rob van der Goot, Lanwenn ar C’horr, Karan Dua, Andrew Yates, Mithil Bangera, Yeshil Bangera, Hitesh Laxmichand Patel, Shu Okabe, Fenal Ashokbhai Ilasariya, Dmitry Gaynullin, Genta Indra Winata, Yiyuan Li, Juan Pablo Martínez, Amit Agarwal, Ikhlasul Akmal Hanif, Raia Abu Ahmad, Esther Adenuga, Filbert Aurelian Tjiaranata, Weerayut Buaphet, Michael Anugraha, Sowmya Vajjala, Benjamin Rice, Azril Hafizi Amirudin, Jesujoba O. Alabi, Srikant Panda, Yassine Toughrai, Bruhan Kyomuhendo, Daniel Ruffinelli, Akshata A, Manuel Goulão, Ej Zhou, Ingrid Gabriela Franco Ramirez, Cristina Aggazzotti, Konstantin Dobler, Jun Kevin, Quentin Pagès, Nicholas Andrews, Nuhu Ibrahim, Mattes Ruckdeschel, Amr Keleg, Mike Zhang, Casper Muziri, Saron Samuel, Sotaro Takeshita, Kun Kerdthaisong, Luca Foppiano, Rasul Dent, Tommaso Green, Ahmad Mustapha Wali, Kamohelo Makaaka, Vicky Feliren, Inshirah Idris, Hande Celikkanat, Abdulhamid Abubakar, Jean Maillard, Benoît Sagot, Thibault Clérice, Kenton Murray, Sarah Luger Abstract Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heterogeneous web data often used to train multilingual language models. In this paper, we introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages. Many of the included languages have been previously under-served, making CommonLID a key resource for developing more representative high-quality text corpora. We show CommonLID’s value by using it, alongside five other common evaluation sets, to test eight popular LID models. We analyse our results to situate our contribution and to provide an overview of the state of the art. In particular, we highlight that existing evaluations overestimate LID accuracy for many languages in the web domain. We make CommonLID and the code used to create it available under an open, permissive license. 2 2 footnotetext: Correspondence to [email protected] 1 Introduction Language technologies should be useful tools for speakers of all languages, but this is far from the case. For example, even the most powerful language models only work well for a small number of languages (namely English and Mandarin Chinese). This is largely due to a lack of available high-quality datasets for many languages ( 32 ; 19 ) . Language identification ( LID ) is one of the first steps in creating larger and better datasets. Despite claims that LID is “solved” ( 36 , e.g. ), LID performance is still quite low for many languages including in key domains, such as web data. Even in English, LID models perform worse on social media text and non-standard varieties ( 6 ) , which make up a large portion of web-crawl-based pre-training datasets such as those derived from Common Crawl 1 1 1 https://commoncrawl.org/ . In addition, many LID systems either do not explicitly support under-served languages or perform poorly on them. For such languages, a majority of the available training data is religious text ( 19 ) , meaning that LID models covering these languages are expected to work on both domains and registers on which they were not trained. This limits practitioners’ ability to make use of web data as a valuable potential source of training examples, since under most approaches, LID systems will continue to perform poorly on web data for under-served languages. This in turn limits the development of language technologies such as large language models for languages other than English (LOTE), especially low-resource languages. Improving LID models would allow us to improve datasets and in turn improve language model performance for many languages. To help address these problems, we introduce CommonLID , a crowd-sourced LID dataset, which we release under an open permissive license. 2 2 2 Link to dataset on publication. The dataset is composed of line-level annotations by native speakers on web text coming from Common Crawl, created in collaboration with over eighty annotators. 3 3 3 All annotators who annotated more than 100 documents were invited to be co-authors on this paper. The dataset includes 109 language varieties, 78 of which contain at least 100 lines. CommonLID is intended as a LID benchmark to highlight where current datasets may be overestimating performance in low-resource, heterogeneous and noisy web contexts. This will help develop more robust models. To facilitate future research in this topic, we also provide a comprehensive comparison of eight of the most common LID models on LID across multiple domains, including web data. We show that LID is far from solved: using domain-specific evaluation datasets, we see that no existing model performs well across all domains. Developing better LID systems is essential for reducing crosslingual inequity in the field. To do this, we need better domain-appropriate LID evaluation datasets for more languages. 2 Issues Affecting LID datasets 2.1 Training Datasets Issues with the datasets used to train LID systems allows us to identify problems in the resulting systems. By identifying weaknesses in LID systems, we can design targeted evaluations for the development of better models. Lack of Open Data The datasets used to train the most popular LID systems (e.g. CLD2 ( 45 ) , CLD3 ( 43 ) , FUN-LangID ( 15 ) , and fasttext) are proprietary and therefore not publicly available. Other models (e.g. 40 ) do not release training data due to legal concerns. Whilst there are models which make their data available (e.g. Franc ( 47 ) , GlotLID ( 27 ) , and OpenLID ( 8 ) ), there is still a lack of open data for training SOTA LID models, especially in the long tail of language coverage. Language Coverage Many of the most popular LID models do not support more than approximately 200 languages, reflecting the coverage of most existing multilingual datasets: e.g. Universal Dependencies ( 37 ; 38 ) and Europarl ( 31 ; 46 ) . Newer corpora have been released aimed at combatting this. One example is The Corpus of Global Language Use ( 18 ) , which provides increased representation to regions such as South Asia, Sub-Saharan Africa, and Oceania. Smol ( 14 ) is another recent effort, covering 123 languages which are rarely represented in other datasets. There have also been efforts to develop region-specific datasets, such as AfroLID ( 3 ; African languages) and 5 (Germanic languages). This can offer targeted improvements for languages in these regions. Domain and Register For LID systems that cover over a few hundred languages, the primary source of data for most low-resource languages comes from religious texts: for example, the Parallel Bible Corpus ( 35 ) which covers over 1000 languages, and MIL-TALE ( 7 ) which covers 2110 languages with the data sourced largely from religious text. Since religious data is all that is available for training for many languages in the so-called

Read the complete text