Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translation
Sarubi Thillainathan, Songchen Yuan, En-Shiun Annie Lee, Sanath Jayasena, Surangika Ranathunga · 2025
Read Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translation by Sarubi Thillainathan, Songchen Yuan, En-Shiun Annie Lee, Sanath Jayasena, Surangika Ranathunga (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22582.txt.
Introduction
Sarubi Thillainathan and four co-authors improve machine translation for languages with little training data. Their twenty twenty-five paper works on Sinhala, Tamil, and English, in special domains with very small data sets. Plain fine-tuning of a multilingual model struggles when data is this scarce.
The team adds two steps. First, more training on domain text in one language at a time, to fill gaps for the weak languages. Second, a transfer step that trains on both in-domain and out-of-domain paired text. Across six language pairs, these steps raised the BLEU score by a clear margin over plain tuning. An ensemble of models added a bit more.
Catalog record
| CREATOR | Sarubi Thillainathan, Songchen Yuan, En-Shiun Annie Lee, Sanath Jayasena, Surangika Ranathunga |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22582
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translation Sarubi Thillainathan, Songchen Yuan, En-Shiun Annie Lee, Sanath Jayasena, Surangika Ranathunga Abstract . Fine-tuning multilingual sequence-to-sequence large language models (msLLMs) has shown promise in developing neural machine translation (NMT) systems for low-resource languages (LRLs). However, conventional single-stage fine-tuning methods struggle in extremely low-resource NMT settings, where training data is very limited. This paper contributes to artificial intelligence by proposing two approaches for adapting msLLMs in these challenging scenarios: (1) continual pre-training (CPT), where the msLLM is further trained with domain-specific monolingual data to compensate for the under-representation of LRLs, and (2) intermediate task transfer learning (ITTL), a method that fine-tunes the msLLM with both in-domain and out-of-domain parallel data to enhance its translation capabilities across various domains and tasks. As an application in engineering, these methods are implemented in NMT systems for Sinhala, Tamil, and English (six language pairs) in domain-specific, extremely low-resource settings (datasets containing fewer than 100,000 samples). Our experiments reveal that these approaches enhance translation performance by an average of +1.47 bilingual evaluation understudy (BLEU) score compared to the standard single-stage fine-tuning baseline across all translation directions. Additionally, a multi-model ensemble further improves performance by an additional BLEU score. 1. Introduction Neural Machine Translation (NMT) systems have achieved remarkable success in recent years, with significant improvements in translating both high-resource and low-resource languages. A promising approach that has emerged to improve the translation of these diverse language pairs is the fine-tuning (single-stage) Large Language Models, particularly multilingual sequence-to-sequence Large Language Models (msLLMs) such as mBART ( Liu et al., 2020 ; Tang et al., 2021 ) and mT5 ( Xue et al., 2021 ) . This technique involves fine-tuning pre-trained msLLMs directly on the language pairs, leveraging their extensive pre-trained knowledge to deliver substantial performance gains compared to training an NMT system from scratch with randomly initialized weights. Studies ( Liu et al., 2020 ; Tang et al., 2021 ; Thillainathan et al., 2021 ; Lee et al., 2022 ) have demonstrated that this method significantly boosts translation accuracy, making it a viable option for improving NMT outcomes. However, while single-stage fine-tuning has shown impressive results, it often falls short for Low-Resource Languages (LRLs) 1 1 1 To define the extent of resource scarcity, we adopt the threshold proposed by ( Ranathunga et al., 2022 ) , classifying NMT’s training data as low-resource or extremely low-resource when the available parallel corpus contains fewer than 0.5 million or 0.1 million sentences, respectively. Nonetheless, it is important to note that this threshold is not an absolute criterion. , particularly those that are underrepresented in the pre-trained msLLMs due to limited monolingual data. This problem is more notable in domain-specific translation. As demonstrated by ( Lee et al., 2022 ) , a straightforward, single-stage fine-tuning of msLLMs tends to yield suboptimal outcomes for these LRLs, requiring more advanced strategies to improve translation performance in such contexts. Moreover, even generative LLMs, continue to struggle significantly with the translation ( Xu et al., 2024 ) . The performance gap between these models and msLLMs remains particularly wide in LRL settings. This underscores the ongoing need for refined training approaches that better accommodate the nuances and scarcities associated with LRL. In this study, we aim to address the existing limitations in the performance of msLLMs, particularly in domain-specific settings, by answering the following research questions: (1) How can the translation performance of msLLMs for LRLs be enhanced by addressing the underrepresentation of these languages in the training data? (2) Beyond conventional single-stage fine-tuning, what alternative techniques can be employed to leverage parallel corpora from different domains to enhance the adaptability and accuracy of msLLMs in translating LRLs? (3) Considering these advanced techniques, how can we develop a comprehensive strategy to significantly improve the performance of msLLMs in translating LRLs in domain-specific settings within NMT frameworks? To answer the question (1) and (2), we investigate two techniques that utilize additional datasets to assist in the training process - Continual Pre-Training (CPT) 2 2 2 CPT—which entails further pre-training of an LLM with monolingual data, refer to Section- 6.3 for more details and Intermediate Task Transfer Learning (ITTL) 3 3 3 ITTL-fine-tuning with the msLLMs with auxiliary parallel data before fine-tuning with the limited target domain data, refer to Section- 6.2 . For question (3), we combined the aforementioned CPT and ITTL two techniques to advance msLLMs for LRL-NMT in domain-specific contexts: (1) Contiually Pre-Train msLLMs with domain-specific monolingual corpora, followed by (2) Fine-Tuning these CPT-enhanced msLLMs using in- and out-domain parallel corpora with ITTL. We propose two ITTL fine-tuning strategies (a) a three-stage bilingual fine-tuning method, involving sequential fine-tuning with out-domain, mixed-domain, and in-domain parallel data (b) a hybrid multilingual and bilingual fine-tuning approach, combining multilingual fine-tuning followed by bilingual fine-tuning. (3) This methodology is further enhanced by the use of ensemble strategies to optimise results. We selected Sinhala, Tamil, and English in extremely LRL domain-specific settings. Among these three languages, we experiment with the possible combination of six language pair translation directions. Our proposed CPT with ITTL strategies outperforms the baseline (direct fine-tuning) strongly for all the six directions where we obtained +2.6 , +1.83 , +1.41 , +1.17 , +1.03 and +0.8 BLEU score improvement respectively for Ta\rightarrow En , Ta\rightarrow Si , Si\rightarrow Ta , Si\rightarrow En , En\rightarrow Ta over strong direct fine-tuning baseline. Furthermore, our multi-model ensemble improves the performance even further by an average of (+2.13) BLEU score. 2. Methodology 2.1. Overview Figure 1. Overview of Methodology We use two single-stage fine-tuning baselines- Bilingual Fine-Tuning (B-FT) and Multilingual Fine-Tuning (M-FT) as described in ( Tang et al., 2021 ) . In B-FT we fine-tune one language pair at a time with domain-specific parallel data. Meanwhile, in M-FT, we fine-tune more than one language pair simultaneously with domain-specific parallel data. M-FT is further categorized into three types: (1) Many-to-One Multilingual Fine-Tuning (M2O-FT): Many\rightarrow Language X (2) One-to-Many Multilingual Fine-Tuning (O2M-FT): Language X\rightarrow Many (3) Many-to-Many Multilingual Fine-Tuning (M2M-FT): Many\rightarrow Many , with Language X as a pivot language. In both B-FT and M-FT, training of the NMT model starts with the weights of the msLLM instead of random initialization. In other words, both the encoder and the decoder of the NMT model are initialized with the weights of the msLLM. The overview of this fine-tuning process is described in Fig. 1 . We investigate CPT and ITTL to obtain better results over single-stage fine-tuning of msLLMs for domain-specific LRL-NMT. These are the main three approaches: • Continual Pre-Training (CPT) of the msLLM for domain adaptation. • ITTL fine-tuning of msLLM with in- and out-domain parallel data for the NMT task. • Ensembling different fine-tuned models. Fig. 1 summarises how our techniques are implemented/related. 2.2. Continual Pre-training for Domain Adaptation To mitigate the under-representation of LRLs within the msLLM, we extend the pre-training phase of the msLLM by integrating supplementary monolingual data. During this CPT process, the msLLM is exposed to a procedure wherein the monolingual data is deliberately ‘noised’, or altered, and then presented as the input (source). The model is then tasked with reconstructing or translating this noised input back into its original, unaltered form