Improving Informally Romanized Language Identification
Adrian Benton, Alexander Gutkin, ∘ Christo Kirov, Brian Roark, Google Research, New York; ∘ London, UK; Portland, OR · 2025
Read Improving Informally Romanized Language Identification by Adrian Benton, Alexander Gutkin, ∘ Christo Kirov, Brian Roark, Google Research, New York; ∘ London, UK; Portland, OR (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21540.txt.
Introduction
Adrian Benton, Alexander Gutkin, Christo Kirov, and Brian Roark improve language spotting in informal Roman-letter text, in a paper from two thousand twenty-five. Many languages normally written in other scripts are typed in Latin letters online, with wild spelling swings. Look-alike languages, such as Hindi and Urdu, then blur together.
The team built training sets that mimic natural spelling variation. A simple linear model trained only on such fake samples beat a large pretrained neural model by a wide margin. Adding real harvested text pushed scores higher still. Twenty Indic languages were covered in the test set.
Catalog record
| CREATOR | Adrian Benton, Alexander Gutkin, ∘ Christo Kirov, Brian Roark, Google Research, New York; ∘ London, UK; Portland, OR |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-21540
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Informally Romanized Language Identification Adrian Benton, Alexander Gutkin, ∘ Christo Kirov, Brian Roark, Google Research, New York; ∘ London, UK; Portland, OR Abstract The Latin script is often used to informally write languages with non-Latin native scripts. In many cases (e.g., most languages in India), the lack of conventional spelling in the Latin script results in high spelling variability. Such romanization renders languages that are normally easily distinguished due to being written in different scripts – Hindi and Urdu, for example – highly confusable. In this work, we increase language identification (LID) accuracy for romanized text by improving the methods used to synthesize training sets. We find that training on synthetic samples which incorporate natural spelling variation yields higher LID system accuracy than including available naturally occurring examples in the training set, or even training higher capacity models. We demonstrate new state-of-the-art LID performance on romanized text from 20 Indic languages in the Bhasha-Abhijnaanam evaluation set Madhani et al. (2023a) , improving test F1 from the reported 74.7% (using a pretrained neural model) to 85.4% using a linear classifier trained solely on synthetic data and 88.2% when also training on available harvested text. 1 Introduction Web crawls are an important source of multilingual training data for natural language modeling, containing rich and diverse spontaneous language, albeit alongside less useful content such as boilerplate and non-language text. Due to this latter noise, to reach adequate levels of data quality, aggressive filtering is typically applied, including setting thresholds on language identification (LID) confidence to retain text ( Raffel et al., 2020 ; Xue et al., 2021 ; Abadji et al., 2022 ; Kudugunta et al., 2023 ; Kargaran et al., 2024 , see, e.g.,) . Filtering text by LID confidence places languages that are less confidently identified at a higher risk of being filtered out, and, indeed, certain classes of text are heavily underrepresented in such collections as a result, such as the romanized text detailed below. This can become a self-reinforcing status, to the extent that such collections may be used as sources of training data for subsequent LID systems. For example, informal romanization – the use of the Latin script without orthography to write languages that are natively written with non-Latin scripts – is both common, e.g., in the languages of South Asia, and generally makes the languages much more difficult to identify. The 22 scheduled languages of India all use non-Latin scripts in their official writing systems, but are also commonly written informally in the Latin script Brandt (2020) . Even so, romanized texts in these languages are mostly omitted from large, multilingual web corpora such as multilingual C4 (mC4, Xue et al. 2021 ), 1 1 1 mC4 contains romanized text in 6 languages that natively use other scripts, but only one South Asian language was included (Hindi), and quality assessments of the romanized sets in mC4 have been unfavorable ( Kreutzer et al., 2022 ) . and are relatively sparse in MADLAD-400 Kudugunta et al. (2023) 2 2 2 https://huggingface.co/datasets/allenai/MADLAD-400 , data released under CC-BY-4.0 license. and GlotCC Kargaran et al. (2024) . 3 3 3 https://github.com/cisnlp/GlotCC , data released under CC0-1.0 license. See Table 3 for data sizes. The distinction between LID system performance on text in the native scripts of languages versus romanized text in those languages is strikingly demonstrated in the Bhasha-Abhijnaanam (denoted B-A) LID benchmark Madhani et al. (2023a) 4 4 4 https://huggingface.co/datasets/ai4bharat/Bhasha-Abhijnaanam , data released under CC0 license. for the 22 scheduled languages of India. The best systems reported in that paper achieved nearly 99% accuracy on the native script LID task, while the best reported performance on the romanized task (covering 20 of the 22 languages) reached just over 80% accuracy. Further, to achieve this latter result, they relied on a relatively expensive BERT model ( Devlin et al., 2019 ) ; a simple fastText Joulin et al. (2016) ; Joulin et al. (2017) linear model yielded the best performance on native script LID, but on the romanized task its performance fell far behind the best system at just 71.5% accuracy. Given the dearth of natural, collected training data in the Latin script for these languages, the above romanized LID results were achieved using synthesized training data created by automatically romanizing native script training sets using a neural sequence-to-sequence transliteration model Madhani et al. (2023b) . In this paper, we evaluate methods for improving romanized LID system performance, including synthesizing training data in a way that mimics natural spelling variation in romanized text; and augmenting synthesized data with distantly supervised datasets (MADLAD-400 and GlotCC). Through careful controlled experimentation, we demonstrate that: • Synthesizing training data with spelling variation by sampling from romanization models provides very large accuracy improvements for this task, even when using smaller, less accurate romanization models; • These improvements are obtained to a larger extent by lightweight linear models, so that higher capacity pretrained models are not needed to reach the best reported results; and • Augmenting the training data by including diversely synthesized copies of the training set and/or some independently harvested data yields further modest improvements. Our best system reduces the best previously reported error rate on this task by over 60% relative. 5 5 5 Explicit system details beyond those provided in this paper along with any new resources and scripts required to replicate these results are available at https://github.com/google-research/google-research/tree/master/informally_romanized_lang_id . We also provide extensive analysis of language confusions, system errors and the kinds of spelling variation produced by our synthesis methods. 2 Background and preliminaries 2.1 Language identification (LID) systems LID is a task with a long history, dating from at least Mustonen (1965) . Jauhiainen et al. (2019) ’s survey notes that LID is typically cast as a text classification task over a closed set of mutually exclusive classes. 6 6 6 Multilingual text, which would seem to contradict this idea of mutual exclusivity, is composed of shorter monolingual text spans for which a single tag per span would suffice. They also note the early adoption of character n -grams as features in diverse LID classification approaches Church (1986) ; Beesley (1988) ; Cavnar and Trenkle (1994) , which remains a key feature set for modern LID Lui and Baldwin (2012) ; Brown (2014) ; Kargaran et al. (2023) ; Burchell et al. (2023) and dialect identification systems Çöltekin et al. (2018) ; Baimukan et al. (2022) . Document level LID assigns labels to entire documents, within which there is typically redundant evidence, hence the classification task is easier. Shorter samples, including sub-sentential strings, present more of a challenge, but such systems can be applied in more scenarios than document-level systems, including, e.g., identifying different language spans in multilingual documents. As stated in the introduction, LID systems are often used for analyzing or filtering large text collections, hence efficient inference is a key consideration. Generally speaking, feature-based linear classifiers are practical alternatives to more computationally expensive neural methods Toftrup et al. (2021) ; Adebara et al. (2022) , including those that might be based on pretrained language models Mohan et al. (2023) ; Manukonda and Kodali (2025) . For challenging scenarios, such as when two languages are highly confusable, more expressive classifiers