Improving Retrieval-Augmented Neural Machine Translation with Monolingual Data
Maxime Bouthors, Josep Crego, François Yvon · 2025
Read Improving Retrieval-Augmented Neural Machine Translation with Monolingual Data by Maxime Bouthors, Josep Crego, François Yvon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21747.txt.
Introduction
Maxime Bouthors, Josep Crego, and François Yvon improve retrieval-augmented machine translation using text in the target language alone, in a research paper from 2025. Such systems usually search a bilingual memory for helpful past translations. But in many settings, only target-language text is plentiful.
Their retriever searches that monolingual text directly, guided by the source sentence. They train it with both sentence-level and word-level matching goals. Across three system designs, the approach matched the quality of memory-based models in controlled tests. On a larger real-world setup, it beat both the plain baseline and general-purpose cross-language retrievers by a clear margin.
Catalog record
| CREATOR | Maxime Bouthors, Josep Crego, François Yvon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-21747
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Retrieval-Augmented Neural Machine Translation with Monolingual Data Maxime Bouthors, Josep Crego, François Yvon Abstract Conventional retrieval-augmented neural machine translation (RANMT) systems leverage bilingual corpora, e.g., translation memories (TMs). Yet, in many settings, in-domain monolingual target-side corpora are often available. This work explores ways to take advantage of such resources by retrieving relevant segments directly in the target language, based on a source-side query. For this, we design improved cross-lingual retrieval systems, trained with both sentence level and word-level matching objectives. In our experiments with two RANMT architectures, we first demonstrate the benefits of such cross-lingual objectives in a controlled setting, obtaining translation performances that surpass standard TM-based models. We then showcase our method on a real-world set-up, where the target monolingual resources far exceed the amount of parallel data and observe large improvements of our new techniques, which outperform both the baseline setting, and general-purpose cross-lingual retrievers. 1 Introduction The use of Retrieval-Augmented Generative Models ( Li et al. 2022 ) is rapidly expanding, owing to their built-in ability to condition generations with illustrative segments retrieved from memory. The combination of (machine) retrieval and (human) regeneration has long been a key principle of Computer Aided Translation (CAT) systems ( Arthern 1978 ; Kay 1997 ; Bowker 2002 ) : segments resembling the source sentence are first retrieved from Translation Memories (TM), providing translators valuable suggestions that can then be edited into a new translation. Such techniques have been transposed in Statistical MT ( Koehn and Senellart 2010 ) , more recently in Retrieval-Augmented Neural Machine Translation (RANMT) e.g., ( Gu et al. 2018 ) , making the entire process fully automatic. Most subsequent work has continued to leverage parallel data in the form of TMs, performing retrieval in the source language, then using the retrieved target side text in generation. Similar principles underlie few-shot MT with large language models, where both the source and target sides of examples are inserted into the prompt Brown et al. 2020 . TM-based approaches typically retrieve examples with lexical fuzzy matchers , using BM25 and/or the Levenshtein distance (LED) as filters or rankers ( Bouthors et al. 2024 ) . These are known to be computationally efficient, and to often surpass continuous-space retrievers ( Xu et al. 2020 ) : this is because lexical (source) matches often come with appropriate translation of terms. However, relying on source-side lexical similarity raises two issues that can cause irrelevant segments to be retrieved: (a) noisy alignments between source and target side of a parallel instance; (b) intrinsic morphosyntactic divergences between the two languages ( Dorr 1994 ) . These problems can be sidestep by performing retrieval directly on the target side , using Cross-Lingual Information Retrieval (CLIR) techniques. An important additional benefit would be to dispense with the use of parallel data and to potentially inform the generation process with relevant monolingual resources. As demonstrated by the recurrent use of target side monolingual data thanks to back-translation Sennrich et al. 2016 , such resources are much easier to find than parallel ones in many domains, and also devoid of so-called translationese phenomena Bogoychev and Sennrich 2020 . As such, monolingual corpus constitute a very useful, yet under-studied, source of high-quality segments for RANMT. To search such resources, cross-lingual retrievers are readily available thanks to multilingual sentence encoders such as LASER ( Artetxe and Schwenk 2019b ) or LaBSE ( Feng et al. 2022 ) . They however lack the critical ability to ensure that the retrieved examples will have a large lexical overlap with the source text. In this work, we propose novel methods to boost the lexical matching abilities of language-agnostic sentence encoders, thereby improving their effectiveness for RANMT. For this, we consider various ways to fine-tune a pre-trained encoder into reproducing the behaviour of lexical fuzzy matchers. We experiment with two language pairs (English-French and German-English) and multiple domains of various sizes and diversities. We consider two types of RANMT systems: an autoregressive source-augmented transformer ( Bulte and Tezcan 2019 ) and a non-autoregressive edit-based model ( Bouthors et al. 2023 ) . We first study the performance of CLIR techniques in a controlled setting, showing that they match standard TM-based models with source-side retrieval. We then consider a much larger set-up, where the target monolingual resources far exceed the amount of parallel data and observe large improvements using our new techniques, which outperform both the baseline setting, as well as generic cross-lingual retrievers. 2 Related Work Our work falls under the scope of Retrieval-Augmented Generation, leveraging both lexical (BM25 ( Robertson and Walker 1994 ) , Levenshtein distance ( Levenshtein 1965 ) ) and semantic methods for the Information Retrieval (IR) component. Semantic methods usually rely on dual encoders trained on parallel data ( Gillick et al. 2018 ) with a contrastive loss ( Sohn 2016 ) . Our focus is on massively multilingual sentence encoders that can encode mutual translations as close neighbors, such as e.g, LASER ( Artetxe and Schwenk 2019b ) and LaBSE ( Feng et al. 2022 ) . In recent years, there has been a growing focus on integrating retrieval components into NMT systems. One key motivation is to enhance model transparency by providing users with the retrieved examples that inform the generated output ( Rudin 2019 ) . A simple implementation of this idea encodes the target side of the example(s) along with the source sentence, providing an additional translation context ( Bulte and Tezcan 2019 ; Gu et al. 2018 ; Xia et al. 2019 ; He et al. 2021 ; Cheng et al. 2022 ; Agrawal et al. 2023 ) . It is also possible to leverage both the source and target sides of the example(s) ( Pham et al. 2020 ; Reheman et al. 2023 ) . Instead of regenerating a complete text with an extended context, edit-based models aim to perform minimal changes to the retrived examples and generate their output in a non-autoregressive (NAR) fashion. Such techniques are studied in ( Niwa et al. 2022 ; Xu et al. 2023 ; Zheng et al. 2023 ) , adapting the Levenshtein transformer ( Gu et al. 2019 ) . This approach is extanded to handle multiple examples that are simultaneously edited in ( Bouthors et al. 2023 ) . MT systems based on Large Language Models (LLMs) can also seamlessly accomodate examples though In-Context Learning (ICL), where the LLM prompt is enriched with multiple demonstrations of the translation task, comprising both the source and target sides of parallel sentences ( Radford et al. 2019 ) . Several studies have focused on optimizing the performance of such architectures for NMT, examining the impact of prompt modifications, the number of demonstrations and the retrieval procedure ( Moslem et al. 2023 ; Vilar et al. 2023 ; Zhang et al. 2023 ; Hendy et al. 2023 ; Bawden and Yvon 2023 ; Zebaze et al. 2024 ) . Our lexical alignment objective corresponds to a form of neural metric learning, where a high dimensional distance space is reduced to a smaller one ( Suárez et al. 2021 ) . This field of research aims to learn distances from the data ( Kulis 2013 ; Bellet et