Ensemble Self-Training for Unsupervised Machine Translation
Ido Aharon, Jonathan Shaki, Sarit Kraus · 2026
Read Ensemble Self-Training for Unsupervised Machine Translation by Ido Aharon, Jonathan Shaki, Sarit Kraus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17087.txt.
Introduction
Aharon, Shaki, and Kraus present an ensemble self training method for unsupervised machine translation. Their models share one primary language pair but differ in an auxiliary language, which creates structured diversity. They generate pseudo translations for the primary pair with token level ensemble decoding, averaging predictions in both directions. These outputs become synthetic parallel data that retrain each model. At deployment, one model is picked by validation score, so inference cost stays single model. Across tests, gains averaged 1.7 chrF from English and 0.67 chrF into English, both statistically significant.
Catalog record
| CREATOR | Ido Aharon, Jonathan Shaki, Sarit Kraus |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38036 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17087
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ensemble Self-Training for Unsupervised Machine Translation Ido Aharon, Jonathan Shaki, Sarit Kraus Abstract We present an ensemble-driven self-training framework for unsupervised neural machine translation (UNMT). Starting from a primary language pair, we train multiple UNMT models that share the same translation task but differ in an auxiliary language, inducing structured diversity across models. We then generate pseudo-translations for the primary pair using token-level ensemble decoding, averaging model predictions in both directions. These ensemble outputs are used as synthetic parallel data to further train each model, allowing the models to improve via shared supervision. At deployment time, we select a single model by validation performance, preserving single-model inference cost. Experiments show statistically significant improvements over single-model UNMT baselines, with mean gains of 1.7 chrF when translating from English and 0.67 chrF when translating into English. 1 Introduction Neural machine translation (NMT) performs well when large parallel corpora are available, but many language pairs—especially low-resource ones—lack such supervision ( 42 ; 11 ; 22 ; 34 ) . Unsupervised neural machine translation (UNMT) addresses this setting by learning from monolingual data alone ( 3 ; 34 ) . Despite substantial progress through back-translation, denoising objectives, and multilingual pretraining ( 26 ; 8 ; 5 ) , UNMT remains fragile: training quality is highly sensitive to initialization, and self-training can reinforce poor pseudo-translations rather than correct them ( 28 ; 21 ; 25 ; 1 ; 18 ) . The central mechanism in UNMT is back-translation ( 11 ; 34 ) , which turns monolingual sentences into synthetic parallel data by translating in one direction and training the reverse direction to reconstruct the original sentence. This enables learning without gold bitext, but it also creates a closed feedback loop: each model is trained on pseudo-labels that it generated itself, or that were generated by a closely related counterpart. When those pseudo-translations are noisy or degenerate, errors and biases can compound over training ( 40 ; 38 ; 43 ) . In difficult settings, such as weak cross-lingual alignment or limited target-language coverage, individual models may converge to poor equilibria rather than to robust translation behavior ( 28 ; 21 ; 25 ) . In this work, we address this limitation by replacing single-model self-training with ensemble-driven self-training . Our key idea is to use multiple UNMT models to generate pseudo-parallel data jointly, rather than relying on any one model’s outputs alone. To obtain meaningful diversity, we train several models on the same primary language pair (A,B) , but assign each model a distinct auxiliary language C_{i} . Each model, therefore, learns the same core translation task while being exposed to different cross-lingual signals. We then ensemble these models at the token level to generate pseudo-translations for the primary pair, and use those ensemble outputs as supervision for further training. In this way, the ensemble is not merely an inference-time trick: it becomes a mechanism for improving the training signal itself. Our framework separates training-time collaboration from deployment-time efficiency . Multiple models collaborate only while generating synthetic supervision; after training, we deploy a single model selected by validation performance. This preserves the inference cost of standard UNMT while allowing training to benefit from more stable and higher-quality pseudo-labels. We evaluate the method in fully unsupervised translation settings spanning both low-resource and high-resource scenarios. In addition, when training decoder-only LLMs for UNMT from weak initialization, we identify two practical failure modes: (1) the model copies the input instead of translating it, and (2) the model produces nearly the same output distribution regardless of the input. To mitigate these issues, we introduce a mixed-training strategy that stabilizes early learning. Across experiments, ensemble-driven self-training yields consistent gains over single-model baselines, with the largest improvements occurring in the more difficult translation direction. These results suggest that structured model diversity and shared pseudo-supervision provide a simple and effective way to improve UNMT without increasing deployment-time complexity. Our contributions are as follows: • We analyze collapse modes that arise when training next-token-prediction LLMs for unsupervised translation from weak initialization, and introduce a mixed-training strategy that stabilizes learning ( Section 4.1 ). • We propose an ensemble-driven self-training framework for UNMT in which multiple models learn from ensemble-generated pseudo-parallel data while differing only in their auxiliary language ( Section 2 ). • We show that ensemble-generated training yields consistent improvements over single-model UNMT baselines while retaining single-model inference efficiency ( Section 5.2 ). 1.1 Related Work UNMT with Decoder-Only LLMs Early UNMT methods relied on architectures and objectives designed specifically for unsupervised translation ( 2 ; 3 ) . More recent work has explored using standard generative LLMs for translation with little or no parallel supervision ( 17 ; 13 ; 5 ) . This shift makes it possible to build UNMT systems directly on top of broadly available pretrained models, which is the setting we consider here. Ensemble Learning Ensembling has recently been studied extensively for LLMs ( 9 ; 30 ; 4 ) . Common approaches include token-level aggregation of predictive distributions ( 19 ; 47 ) , routing inputs to specialized models ( 48 ; 7 ) , averaging weights across models ( 39 ) and majority-style answer selection after reasoning ( 24 ; 9 ) . In unsupervised translation, however, routing is nontrivial and explicit reasoning is typically absent, making token-level ensembling the most natural choice. Ensemble Methods for Machine Translation Ensemble methods have shown gains in supervised NMT ( 15 ; 44 ; 27 ) , but, to our knowledge, they have not been used as a training-time supervision mechanism in UNMT. This distinction is important: in supervised settings, ensembles are typically used at inference time, whereas in our setting, ensemble outputs can be recycled into additional synthetic supervision. Our method leverages this property while still avoiding multi-model deployment. Evaluation We evaluate translation quality primarily using chrF ( 32 ) , a character n-gram F-score that has been shown to correlate well with human judgments ( 41 ; 12 ; 29 ) , particularly for morphologically rich languages. In addition, we report BLEU ( 31 ; 36 ; 37 ) scores using sacreBLEU ( 33 ) 1 1 1 Signature: nrefs:1|case:mixed|eff:no|tok:13a| smooth:exp|version:2.5.1 . to facilitate comparison with prior UNMT work ( 17 ; 10 ; 2 ; 3 ; 25 ) . Finally, we report COMET ( 35 ; 16 ) 2 2 2 Using XComet-XXL as evaluator. as a neural evaluation metric that has been shown to better capture semantic adequacy. chrF serves as our primary evaluation metric, while BLEU and COMET are provided as complementary indicators of translation quality. 2 Method 2.1 Overview Our approach proceeds in three stages. First, we independently train several UNMT models that all learn translation for the main language pair (A,B) , but each additionally incorporates a distinct auxiliary language C_{i} . The presence of different auxiliary languages encourages diversity across models. Second, we combine the predictions of these models during decoding using token-level ensembling. The resulting ensemble translations are treated as pseudo-parallel data for the main language pair. Each model is then further trained on this synthetic data, enabling iterative improvement through collective supervision. Finally, we select a single model for deployment based on validation performance. In this