MultiSlav: Using Cross-Lingual Knowledge Transfer to Combat the Curse of Multilinguality
Artur Kot, Mikołaj Koszowski, Wojciech Chojnowski, Mieszko Rutkowski, Artur Nowakowski, Kamil Guttmann, Mikołaj Pokrywka · 2025
Read MultiSlav: Using Cross-Lingual Knowledge Transfer to Combat the Curse of Multilinguality by Artur Kot, Mikołaj Koszowski, Wojciech Chojnowski, Mieszko Rutkowski, Artur Nowakowski, Kamil Guttmann, Mikołaj Pokrywka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14509.txt.
Introduction
Artur Kot and co-authors present MultiSlav, a set of open-source translation models for Slavic languages. Their 2025 paper asks whether multilingual neural machine translation suffers a curse of multilinguality or gains from cross-lingual transfer within a family.
Slavic languages are morphologically rich and spoken by hundreds of millions, yet their machine translation is under-studied. The team explored ways to extend the data available for training. They show cross-lingual benefits even in zero-shot translation for low-resource languages. Their models, released openly under a permissive license, are described as state of the art for the selected Slavic language pairs, giving a resource other researchers can build on.
Catalog record
| CREATOR | Artur Kot, Mikołaj Koszowski, Wojciech Chojnowski, Mieszko Rutkowski, Artur Nowakowski, Kamil Guttmann, Mikołaj Pokrywka |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14509
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MultiSlav: Using Cross-Lingual Knowledge Transfer to Combat the Curse of Multilinguality Artur Kot, Mikołaj Koszowski, Wojciech Chojnowski, Mieszko Rutkowski, Artur Nowakowski, Kamil Guttmann, Mikołaj Pokrywka Abstract Does multilingual Neural Machine Translation (NMT) lead to The Curse of the Multlinguality or provides the Cross-lingual Knowledge Transfer within a language family? In this study, we explore multiple approaches for extending the available data-regime in NMT and we prove cross-lingual benefits even in 0-shot translation regime for low-resource languages. With this paper, we provide state-of-the-art open-source NMT models for translating between selected Slavic languages. We released our models on the HuggingFace Hub 1 1 1 Link: https://hf.co/collections/allegro/multislav-6793d6b6419e5963e759a683 under the CC BY 4.0 license. Slavic language family comprises morphologically rich Central and Eastern European languages. Although counting hundreds of millions of native speakers, Slavic Neural Machine Translation is under-studied in our opinion. Recently, most NMT research focuses either on: high-resource languages like English, Spanish, and German - in WMT23 General Translation Task Kocmi et al. 2023 7 out of 8 task directions are from or to English; massively multilingual models covering multiple language groups; or evaluation techniques. 1 Introduction In the literature, we can find 2 seemingly contradictory observations about multilingual models: (1) adding more languages to NLP models will lead to Cross-lingual Knowledge Transfer increasing the quality of the model, notably for low-resource languages and primarily for related or geographically co-located languages Koloski et al. 2022 ; Adelani et al. 2022 ; (2) adding more languages to the model may lead to The Curse of the Multilinguality , reducing the quality of the model, especially for high-resource languages Conneau et al. 2020 . The rule of thumb is: that only languages from the same language group (and written in the same script) should increase the quality of the model. However, we are not aware of any study validating or disproving this claim for Slavic languages. In this paper, we provide a study of the application of a Multilingual NMT approach to the group of low- and mid-resource (defined in section 3 ) languages represented by selected Latin-script Slavic languages: Czech, Polish, Slovak, and Slovene. We explore the extension of this group by adding the high-resource language - English. The English language is culturally influential in the modern era whilst also providing access to a large number of parallel examples (bitext) for selected languages, increasing the open-source pool by a factor of 3 . Explored strategies are presented in Figure 1 . Language Pairs Data size 1 Czech\leftrightarrow Polish 63M 2 Czech\leftrightarrow Slovak 30M 3 Czech\leftrightarrow Slovene 25M 4 Polish\leftrightarrow Slovak 26M 5 Polish\leftrightarrow Slovene 23M 6 Slovak\leftrightarrow Slovene 18M 7 English\leftrightarrow Czech 151M 8 English\leftrightarrow Polish 150M 9 English\leftrightarrow Slovak 52M 10 English\leftrightarrow Slovene 40M Slavic directions 185M All directions 578M Table 1: Size of open-source training bitext for each pair of languages in Millions of parallel sentences. Size is counted after filtering and deduplication. (a) Bi-Directional Data (baseline 63M examples) (b) Pivot Slavic Data (+55M examples) (c) Multilingual Slavic Data (+122M) (d) Multilingual Slavic + English Data (+515M) Figure 1: Strategies for increasing the data-regime without decreasing the quality of the model illustrated by example of translating from Polish to Czech language. In parenthesis we show how many data points were added compared to baseline. In our study: (1) we evaluate several multilingual translation scenarios: Bi-Directional models, Multi-way Multilingual models ( Many2Many ) Firat et al. 2016 and Pivot models Kim et al. 2019 ; Utiyama and Isahara 2007 and evaluate the quality of the methods for each of the selected Slavic languages; (2) we evaluate how adding English language to the selected group impacts performance of the models; (3) we investigate the impact of the Cross-lingual Knowledge Transfer in a narrow language group. The results of this study confirm the Cross-lingual Knowledge Transfer hypothesis for the translation between Slavic languages. Indeed, multilingual training increases the quality of the lower data-regime direction (e.g. Slovak to Slovene), even in the directional zero-shot Johnson et al. 2017 regime and on the language pair not present in the training data. 2 Related work The underlying system architecture of most of the commercial MT vendors is closed and unknown but released information suggests heavy use of transformers architecture Vaswani et al. 2017 and Many2Many translation models Johnson et al. 2017 . Meta released multiple multilingual MT models, starting from mBART Liu et al. 2020 , M2M Fan et al. 2021 , NLLB NLLB Team et al. 2022 and most recently SeamlessM4T Seamless Communication et al. 2023 . There are also fully open-source initiatives led by the University of Helsinki centered around OPUS corpora collection Tiedemann 2012 with multiple releases of MT models Tiedemann and Thottingal 2020 ; Tiedemann 2020 . Prior research on the cross-lingual knowledge transefer in NMT was conducted for Indic Bala Das et al. 2023 , and Turkic Mirzakhalov et al. 2021 language families. Moreover, Large Language Models (LLMs) have recently entered the scene of Machine Translation. Proprietary LLMs often outperform custom translation models on the high-resource languages Kocmi et al. 2023 , but still lag behind classical solutions in the mid-, and low-resource regime Hendy et al. 2023 ; Zhu et al. 2023 . For extensive reviews on the multilingual machine translation methodologies see Kocmi et al. 2021 and Dabre et al. 2020 . 3 Data 3.1 Data Sources Training datasets were downloaded via MTData library Gowda et al. 2021 , see details in Appendix G and in Table 10 . The aggregated sources, languages supported, and size of each corpus can be found in Table 1 . For evaluation, we use the parallel dataset from Flores 101 - dev Goyal et al. 2022 , which contains 997 sentences translated into multiple languages including all 5 languages in our scope. 3.2 Data Filtering Firstly, we normalize text by removing special characters, unifying quotations and whitespaces, and applying the Unicode NFKC (Normalization Form Compatibility Composition) normalization. Then we filter out potentially misaligned sentences using the following text-based features: (1) Levenshtein distance between source and target sentences; (2) sentence length in characters; (3) sentence length in amount of tokenized words; (4) FastText language detection Joulin et al. 2016 2 2 2 FastText model version: LID.176 ; (5) Poisson-based log-probability for sentence length ratios Koszowski et al. 2021 ; (6) mismatched numbers; (7) ratio of digits to other characters; (8) average length of tokenized words; (9) maximum length of the longest word in the sentence; (10) alphabet-based non-whitelist character ratio to the rest of characters. Duplicate training pairs are removed if either side, source sentence, or target sentence, is already present in the dataset. 4 Experimental Setup In this section, we describe tokenizer training, special tokens for language hinting, and model architecture used for training models. Model Type Directions supported Size baseline (ours) bi-directional All 20 directions 10 models 209M each Google Translate March 2024 All 20 directions N/A PaLM-2 March 2024 All 20 directions N/A GPT-3.5 March 2024 All 20 directions N/A M2M-100 Many2Many All 20 directions 1.2B NLLB-200 Many2Many All 20 directions 1.3B OPUS-MT Sla-Sla^{*} Many2Many 12 directions, missing SLK pairs 64M OPUS-MT SK-EN^{*} Bi Directional 2 directions: SLK\leftrightarrow ENG MultiSlav (ours) Many2Many 12 directions, missing ENG pairs 242M MultiSlav_{+ENG} (ours) Many2Many All