Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets
Iaroslav Chelombitko, Mika Hämäläinen, Aleksey Komissarov · 2026
Read Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets by Iaroslav Chelombitko, Mika Hämäläinen, Aleksey Komissarov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18791.txt.
Introduction
Iaroslav Chelombitko, Mika Hämäläinen, and Aleksey Komissarov set two and a half score of tongues side by side by the parts of their words. Most past tongue work pairs two or a few tongues at a time. That slow pace can't give a world chart of how tongues kin and part. The team build word sets from the free ref work in each tongue, for all tongues that write in two main scripts.
They cut each word set into parts by a learn tool that finds the cuts that shrink the set most. Each tongue then gets a code of how oft each part shows. Codes can be set side by side to grade how much word stock two tongues share. A chart at that scale could show kin ties and drift across whole lands at once.
Catalog record
| CREATOR | Iaroslav Chelombitko, Mika Hämäläinen, Aleksey Komissarov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18791
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets Iaroslav Chelombitko, Mika Hämäläinen, Aleksey Komissarov Abstract We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing ’glottosets’ from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework. Keywords Multilingual NLP \cdot Comparative Linguistics \cdot BPE Tokenization \cdot Cross-linguistic Analysis \cdot Lexical Similarity \cdot Script-based Typology 1 Introduction Traditional comparative linguistics, while providing deep historical and typological insights (see 34 ; 5 ; 36 ; 38 ; 45 ), often lacks the scalability to handle the current volume of digital text ( 3 ; 2 ). However, recent NLP advances facilitate massive, data-driven studies (see 19 ; 6 ; 29 ; 49 ), revealing universal tendencies in phonetics ( 7 ), lexical semantics ( 50 ), and sound symbolism ( 18 ; 11 ) that were previously inaccessible through manual, small-scale methods. Despite this progress, large-scale studies frequently neglect endangered languages ( 27 ) and systematic performance disparities in multilingual models (see 47 ; 14 ; 22 ). We address this gap by examining 242 languages through a novel script-level lens (Latin vs. Cyrillic), moving beyond family-specific comparisons ( 35 ; 5 ) to reveal overarching patterns visible only when languages share a script-based writing system ( 20 ; 44 ). In particular, we adopt a subword-based strategy ( 46 ) using Byte-Pair Encoding (BPE) ( 24 ) tokenizers, which we train both on individual languages and on large aggregated corpora for each script. Aggregating data for all Latin and Cyrillic-script languages into respective training sets effectively unifies each script community into a single model, leveraging the shared-parameter paradigm common in massively multilingual pretraining (see 31 ; 17 ). Figure 1: Pipeline architecture for subword-based comparative linguistics across 242 languages using Wikipedia glottosets. The workflow illustrates the transformation of Wikipedia dumps (320 languages) through sequential stages: script-based filtering yielding 37 Cyrillic and 205 Latin script languages, monolingual glottoset construction with TF/DF metrics, BPE tokenization (both individual and combined training with 4096 tokens vocabulary), and vector-based subword analysis. Each colored node represents a distinct processing phase, culminating in macro-level insights for script-level comparative linguistics. This modular approach enables scalable analysis of morphological patterns across multiple languages simultaneously. Although subword units are not perfect morphemes (see 46 ; 8 ), they serve as robust automated tools for macro-linguistic research (see 33 ; 39 ; 23 ) that scale more efficiently than manual expert alignment (see 10 ; 42 ; 16 ). Leveraging the framework (Figure 1 ), this work makes two key contributions: 1. Script-Level Aggregation: We demonstrate how combining languages by script (Latin vs. Cyrillic) enables macro-level comparative insights difficult to achieve through traditional language-by-language lenses. 2. Practical Subword Methodology: We introduce a tractable, automated framework that reduces reliance on manual annotation by providing data-driven lexical segments for robust cross-linguistic analysis. While traditional comparative linguistics focuses on genetic relationships and historical reconstruction, and typology examines structural similarities regardless of ancestry, our subword-based approach bridges both: we demonstrate that BPE tokenization captures phylogenetic signal (Section 4.3 ) while also revealing typological patterns across unrelated languages sharing the same script. 2 Related Work Comparative linguistics in NLP has evolved from general data-driven frameworks (see 6 ; 49 ; 29 ) to specialized tasks like automated cognate detection ( 16 ). While manual expert annotation remains the high-precision gold standard ( 36 ), 42 demonstrated that automated methods can reconstruct language phylogenies with accuracy closely approximating expert-curated data ( 30 ). These approaches highlight the potential for scaling linguistic analysis across diverse language families where manual examination is impractical. To overcome the scarcity of parallel data in low-resource settings, researchers have utilized neural machine translation (NMT) to infer cognate relationships ( 26 ). This direction has proven particularly effective for endangered Uralic languages, leveraging cross-lingual relations (see 38 ; 13 ; 12 ) and data augmentation through synthetic cognates generated by statistical machine translation (SMT) ( 40 ). Such hybrid strategies address resource limitations while enriching existing linguistic databases for under-represented script communities. More recently, architectures inspired by computational biology have addressed previous computational bottlenecks. 1 introduced transformer-based models that utilize multiple sequence alignments and link prediction components, mirroring techniques used in genomic research. By treating linguistic evolution through an end-to-end lens, these models significantly reduce computation time while maintaining high accuracy ( 9 ), providing a scalable alternative to traditional alignment-based methods. Automated language typology remains another central research vein ( 41 ), with studies using information-theoretic measures to quantify morphological synthesis and fusion (see 43 ; 37 ). Crucially, 25 found that BPE compression ratios directly correlate with morphological complexity. This aligns with broader efforts to leverage cross-lingual representations that capture typological relationships even without parallel data ( 51 ), although the distribution of this linguistic information varies across model layers based on pretraining strategy ( 15 ). 3 Methodology 3.1 Monolingual Glottosets Construction We downloaded Wikipedia dumps in ZIM format for 320 languages available in the ZIM archive 1 1 1 https://dumps.wikimedia.org/kiwix/zim/wikipedia/ . After that, we selected either the 2024 or 2023 version (preferring the 2024 version if available), otherwise, we used the 2023 version. We included all topics by selecting the “all” files option. To clean the data from unnecessary HTML elements, we used a custom script available in our repository. In brief, this script resolves redirects and extracts only paragraph text while removing as many auxiliary HTML tags as possible, leaving only clean paragraph texts. Figure 2: Visualization of interactive BPE merge graphs for Ukrainian (left) and Finnish (right) subword tokenization patterns. The diagrams show directed merge sequences with reuse counts indicated in parentheses. Vertical black bars represent merge steps, while edges show the progression of subword unit formation. The contrasting patterns reflect language-specific morphological characteristics: Ukrainian showing consistent Cyrillic character combinations, while Finnish exhibits agglutinative patterns. Interactive web application available on our repository. The extracted paragraphs were filtered based on script consistency. For Cyrillic, we retained only Cyrillic paragraphs, and for Latin, only those written in the Latin script. Unfortunately, there is currently no reliable tool for accurately detecting all 242 languages, so at this stage, we filtered only by script (Table 1 ). Additionally, an effective filtering step was to remove all paragraphs containing fewer than 10 words. This significantly reduced noise in the dataset, including accidental words from other languages. Table 1: Statistics for top 30 languages by token count from Wikipedia dumps. Columns show language names and codes, number of raw and cleaned paragraphs,