One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models
Benedikt Ebing, Lennart Keller, Goran Glavaš · 2026
Read One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models by Benedikt Ebing, Lennart Keller, Goran Glavaš (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05776.txt.
Introduction
Benedikt Ebing and 2 co-authors test in 2026 whether converting many scripts into Roman letters helps or harms multilingual models from the start. Romanization exposes shared words across tongues and had raised transfer in favorable setups. Whether it suits general-purpose training, above all for well-resourced tongues, was unclear.
The team pretrained encoder models from scratch on romanized and original text for six diverse high-resource languages. Two romanizers with different fidelity were compared. Tongues with segmental scripts lost almost nothing, while Chinese and Japanese, whose scripts encode syllable-sized units, degraded even with the better romanizer. Increased vocabulary overlap caused no measurable negative interference, and romanization actually raised encoding speed for segmental scripts at negligible cost. The upshot is that one script can serve many tongues, as long as the script fits how the tongue builds its words.
Catalog record
| CREATOR | Benedikt Ebing, Lennart Keller, Goran Glavaš |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05776
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models Benedikt Ebing, Lennart Keller, Goran Glavaš Abstract Exposing latent lexical overlap, script romanization has emerged as an effective strategy for improving cross-lingual transfer (XLT) in multilingual language models (mLMs). Most prior work, however, focused on setups that favor romanization the most: (1) transfer from high-resource Latin-script to low-resource non-Latin-script languages and/or (2) between genealogically closely related languages with different scripts. It thus remains unclear whether romanization is a good representation choice for pretraining general-purpose mLMs, or, more precisely, if information loss associated with romanization harms performance for high-resource languages. We address this gap by pretraining encoder LMs from scratch on both romanized and original texts for six typologically diverse high-resource languages, investigating two potential sources of degradation: (i) loss of script-specific information and (ii) negative cross-lingual interference from increased vocabulary overlap. Using two romanizers with different fidelity profiles, we observe negligible performance loss for languages with segmental scripts, whereas languages with morphosyllabic scripts (Chinese and Japanese) suffer degradation that higher-fidelity romanization mitigates but cannot fully recover. Importantly, comparing monolingual LMs with their mLM counterpart, we find no evidence that increased subword overlap induces negative interference. We further show that romanization improves encoding efficiency (i.e., fertility) for segmental scripts at a negligible performance cost. rmTeXGyreTermesX [*devanagari]rm[ Path = ./fonts/, Extension = .ttf, UprightFont = Lohit-Devanagari, Script = Devanagari ]Lohit-Devanagari [*arabic]rm[ Path = ./fonts/, Extension = .ttf, UprightFont = NotoSansArabic-Regular, Script = Arabic ]NotoSansArabic-Regular [russian]rm[ Path = ./fonts/, Extension = .ttf, UprightFont = CharisSIL-Regular, BoldFont = CharisSIL-Bold ]CharisSIL-Regular [japanese]rm[ Extension = .otf, UprightFont = HaranoAjiMincho-Regular ]HaranoAjiMincho-Regular [chinese]rm[ Extension = .otf, UprightFont = FandolSong-Regular ]FandolSong-Regular 1 Motivation and Background Encoder LMs still outperform their comparable-size (or even larger) decoder-only counterparts on many natural language understanding (NLU) tasks, like classification, clustering, or retrieval Weller et al. 2025 ; Gisserot-Boukhlef et al. 2025 ; and training both monolingual Portes et al. 2023 ; Breton et al. 2025 ; Ehrmanntraut et al. 2025 ; Warner et al. 2025 and multilingual encoder LMs has recently regained traction Marone et al. 2025 ; Boizard et al. 2025 . Multilingual LMs (mLMs) have long been used as primary vehicles for downstream cross-lingual transfer (XLT) ( Dufter and Schütze 2020 ; Muller et al. 2021b ) : fine-tuned on task data in a source language, they perform the same task in a target language with few ( Xu and Murray 2022 ; Schmidt et al. 2022a , few-shot XLT ) or no labeled task examples ( Pires et al. 2019 ; Lauscher et al. 2020 ; Schmidt et al. 2023 , zero-shot XLT ) . More recently, romanization —converting non-Latin scripts to the Latin alphabet—has emerged as a tool to break the script barrier in multilingual language learning Ma et al. 2025 ; Xhelili et al. 2024 ; J et al. 2024 . Mapping diverse scripts to a common symbolic representation, romanization exposes latent lexical overlap and thus directly facilitates XLT with mLMs Amrhein and Sennrich 2020 ; Dabre et al. 2022 ; Moosa et al. 2023 . Romanization is particularly suited for encoder LMs, as we can, unlike in text generation with decoders, avoid the non-trivial additional step of re-mapping the romanized text back to the native script. Still, being non-injective, romanization is inherently lossy: multiple native-script symbols may map to the same (sequences of) Latin characters. This creates a trade-off between enhanced cross-lingual transfer through script unification and the potential loss of script-specific information. Prior research on romanization in LMs shows promising results, but studies have predominantly focused on XLT and two scenarios that arguably favor romanization the most: (1) transfer within a language group with known (lexical) similarities, such as genealogically related languages with different scripts (e.g., languages of the Indo-Aryan family) Khemchandani et al. 2021 ; Dhamecha et al. 2021 ; Liu et al. 2025b , or (2) transfer from a high-resource Latin-script language (usually English) to the text-lean languages from the long tail with non-Latin scripts Muller et al. 2021a ; Purkayastha et al. 2023 ; Liu et al. 2024 ; Liu et al. 2025a . These are precisely the setups in which the benefit of unlocking lexical overlap outweighs the drawback of information loss the most . What remains unclear, however, is how romanization affects the high-resource languages, for which training data is abundant and thus (a) the benefits of exposing lexical links to other languages much smaller and (b) total information loss much larger, as it scales with the training data. This gap is critical for assessing whether romanization is a good representation choice for pretraining general-purpose mLMs, which need to exhibit strong in-language performance for high-resource languages as much as robust XLT performance. Focused Contribution. In this controlled study, we address this gap by quantifying the downstream performance gaps for high-resource languages between encoder LMs pretrained on romanized vs. native-script data. We (pre)train from scratch both monolingual LMs, to isolate the impact of the loss of script-specific information, and corresponding mLMs, to assess whether increased vocabulary overlap leads to negative interference. We experiment with six typologically diverse high-resource languages that span six writing systems, and investigate two romanization variants, produced by two popular romanization tools with different fidelity characteristics. Results on five established downstream tasks reveal that romanized LMs show small to negligible performance losses compared to native script counterparts, in most cases within one standard deviation. While the losses are more pronounced for Chinese and Japanese, two of our experimental languages with morphosyllabic scripts, we show that higher-fidelity romanization can partially mitigate them. Finally, we analyze the trade-off between information loss and tokenizer’s fertility gain: we find that romanized models yield substantially better fertility, while performing only marginally worse than their native-script counterparts. 2 Romanization Approaches Most existing work pre-selects the romanization tool. In contrast, to isolate the downstream effects of the romanizer choice, we compare two widely-used romanizers, which give rise to a trade-off between romanization fidelity and script coverage: URoman. URoman Hermjakob et al. 2018 aims to maximize script coverage by combining (i) parsing character descriptions from the Unicode codepoint database to derive phonetic Latin correspondences, with (ii) manual corrections where this heuristic fails and (iii) established script-specific mappings for scripts where Unicode descriptions do not provide reliable cues for Latin equivalents (e.g., Chinese characters). As a result, URoman offers a near-universal, ASCII-constrained romanization. UConv. UConv bundles standardized romanization schemes, incorporating both transliteration and transcription-focused approaches. While its script coverage is narrower than that of URoman , its adherence to standards involves the use of diacritics, thereby enhancing romanization fidelity by increasing phonemic and orthographic distinctions. We provide further details and romanization examples in Appendix A . arb_Arab cmn_Hani hin_Deva jpn_Jpan rus_Cyrl vie_Latn Seq. Avg. Token Avg. Avg. MonoNat 80.3 ±0.6 78.4 ±0.7 81.0 ±1.6 74.9 ±1.2 81.8 ±0.6 82.0 ±0.7 82.2 ±1.0 76.6 ±1.0 79.7 ±1.0 MonoUroman 80.7 ±0.8 74.1 ±1.0 80.7 ±0.8 71.7 ±1.1 81.6 ±0.6 81.9 ±0.7 81.5 ±0.8 74.7 ±0.8 78.4 ±0.8 MonoUconv 80.3 ±0.8 76.3 ±1.0 80.9 ±0.5 73.6 ±0.9 82.0 ±0.5 82.0 ±0.7 81.9 ±0.8 75.8 ±0.8 79.2 ±0.8 MultiNat 77.5 ±1.2 76.5 ±0.6 77.6 ±1.6 72.0 ±1.3 78.8 ±1.2 79.3 ±1.1 79.3 ±1.2 74.0 ±1.2 76.9 ±1.2 MultiRom 77.3 ±1.0 74.5