Calibrating Translation Decoding with Quality Estimation on LLMs
Di Wu Yibin Lei Christof Monz · 2025
Read Calibrating Translation Decoding with Quality Estimation on LLMs by Di Wu Yibin Lei Christof Monz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19044.txt.
Introduction
Di Wu and co-authors fix a flaw in how translation models pick their final answer. Standard decoding takes the model's most likely sentence. Yet the most likely is often not the best, and can even be garbled. Their fix recalibrates the model's scores so they track true quality, judged from a spread wide view.
They train the scores to correlate with quality marks, using only a few thousand examples per language pair. Gains stack on top of normal fine tuning. Even star translation models improved, in machine scores and human grades alike. The tuned score itself doubles as a quality gauge that rivals dedicated checkers. Code and data are released.
Catalog record
| CREATOR | Di Wu Yibin Lei Christof Monz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19044
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Calibrating Translation Decoding with Quality Estimation on LLMs Di Wu Yibin Lei Christof Monz Abstract Neural machine translation (NMT) systems typically employ maximum a posteriori (MAP) decoding to select the highest-scoring translation from the distribution. However, recent evidence highlights the inadequacy of MAP decoding, often resulting in low-quality or even pathological hypotheses as the decoding objective is only weakly aligned with real-world translation quality. This paper proposes to calibrate hypothesis likelihood with translation quality from a distributional view by directly optimizing their Pearson correlation, thereby enhancing decoding effectiveness. With our method, translation with large language models (LLMs) improves substantially after limited training (2K instances per direction). This improvement is orthogonal to those achieved through supervised fine-tuning, leading to substantial gains across a broad range of metrics and human evaluations. This holds even when applied to top-performing translation-specialized LLMs fine-tuned on high-quality translation data, such as Tower, or when compared to recent preference optimization methods, like CPO. Moreover, the calibrated translation likelihood can directly serve as a strong proxy for translation quality, closely approximating or even surpassing some state-of-the-art translation quality estimation models, like CometKiwi. Lastly, our in-depth analysis demonstrates that calibration enhances the effectiveness of MAP decoding, thereby enabling greater efficiency in real-world deployment. The resulting state-of-the-art translation model, which covers 10 languages, along with the accompanying code and human evaluation data, has been released: https://github.com/moore3930/calibrating-llm-mt . 1 Introduction Figure 1: The illustration of quality-aware decoding (top) and our calibration method (bottom). The former explores the decoding space by adding an extra step during test-time decoding, which involves multiple rounds of sampling followed by reranking or voting to select the highest-scoring output. Our method focuses on training-time optimization, aiming to calibrate the likelihoods of hypotheses to their corresponding quality scores, enabling effective approximated MAP decoding. The training of neural machine translation (NMT) has long been formulated as a maximum likelihood estimation (MLE) problem, and maximum a posteriori decoding (MAP) is a common decision rule, aiming to identify the highest-scoring translation. However, recent findings in NMT suggest that such a system has serious flaws. One particularly counterintuitive issue is the beam search curse ( Koehn and Knowles 2017 ; Murray and Chiang 2018 ; Ott et al. 2018 ; Kumar and Sarawagi 2019 ) , where translation quality deteriorates as search approximations improve and higher-probability hypotheses are possibly worse translations. Ott et al. 2018 explain this as a calibration issue. There is a generally low correlation between hypothesis likelihood and quality beyond a certain likelihood value. The flaws of poorly calibrated translation models can be categorized into two main aspects: First, their performance tends to be suboptimal, as better hypotheses may remain hidden within the distribution mass, inaccessible to MAP decoding ( Ott et al. 2018 ) . Second, likelihood, as a key measure of uncertainty, can serve as a valuable indicator of translation errors within the output at test time ( Wang et al. 2020 ; Fomicheva et al. 2020 ) ; however, poorly calibrated models cannot effectively support this use. Prior studies have tried to mitigate this miscalibration issue by introducing an additional optimization step during inference time, as shown in Figure 1 (top), known as quality-aware decoding (QAD) ( Fernandes et al. 2022 ) . These approaches typically involve generating multiple candidate translations through sampling, followed by reranking or voting using reference-free and/or reference-based machine translation metrics, such as Best-of-N (BoN) sampling ( Rei et al. 2024a ; Faria et al. 2024 ; Brown et al. 2024 ; Ichihara et al. 2025 ) and Minimum Bayes Risk (MBR) decoding ( Kumar and Byrne 2004 ; Freitag et al. 2022a ) . While effective, this line of methods inevitably incurs significantly higher latency. For example, the current top-performing system in the WMT24 competition ( Kocmi et al. 2024a ) , Tower ( Rei et al. 2024a ) , requires sampling 100 times for each prompt from a 70B LLM-based translation model, which is impractical, particularly for online environments. This paper proposes a simple yet highly effective method to directly optimize the calibration during training, where we encourage the likelihood of translation hypotheses to align more closely with their quality by directly optimizing Pearson correlations between them. More specifically, as shown in Figure 1 (bottom), given a translation prompt x , we sample multiple hypotheses from an NMT model during training time and compute both their likelihoods and quality scores, as measured by any external metric, such as COMET. We then define the loss as the negative Pearson correlation between the two sets of data points and minimize it directly using a standard gradient-based optimizer. While simple, our approach yields substantial performance improvements with limited training—using only 2K instances per language—even when applied to state-of-the-art LLM-based MT systems such as Tower. Moreover, due to clear calibration improvements, the resulting models’ likelihood can directly serve as a strong proxy for translation quality, even surpassing some state-of-the-art translation quality estimation (QE) models, like Comet-Kiwi ( Rei et al. 2022a ) . Our main contributions are summarized as follows: • We introduce a simple yet effective method to mitigate the well-known miscalibration issue in translation during training time, where we calibrate translation decoding for quality from a holistic view by directly optimizing the Pearson correlation objective. With limited training, our models consistently outperform the strongest LLM-based MT systems, such as ALMA ( Xu et al. 2023 ; Xu et al. 2024 ) and Tower ( Rei et al. 2024a ; Alves et al. 2024 ) series as well as the GPT-4o model, by a substantial margin across multiple automatic and human evaluation metrics. Extensive experiments demonstrate effectiveness over recent preference optimization methods, such as CPO ( Xu et al. 2024 ) , for translation. • Our method, to some extent, unifies quality optimization and quality estimation (QE) in translation by sharing one single objective. Specifically, the uncertainty quantification of our model’s output, measured by average log-likelihood, exhibits a strong correlation with human judgments, which even outperforms some state-of-the-art QE models in the community that were explicitly trained with human-annotated data, such as CometKiwi ( Rei et al. 2022b ) . This finding supports the view that a well-performing model should inherently “know” what constitutes a good translation. • Our in-depth analysis demonstrates that improved likelihood-quality calibration—achieved through our calibration method—enhances the effectiveness of maximum a posteriori decoding, offering greater efficiency potential for real-world deployment. 2 Background At a practical level, this study aims to optimize both translation quality as well as quality estimation, simultaneously in a single model. To this end, we briefly introduce the relevant foundational concepts. Translation metric meta-evaluation. To automatically assess translation quality, various metrics have been proposed over the past decades, such as BLEU ( Papineni et al. 2002 ) , ChrF ( Popović 2015 ) , COMET ( Rei et al. 2020 ; Rei et al. 2022b ) , MetricX ( Juraska et al. 2023 ; Juraska et al. 2024 ) . Metric meta-evaluation assesses how well these metrics correlate with human judgments, where a few well-known statistics are applied, such as Pearson’s r , Spearman’s \rho , and Kendall’s \tau . In this paper, we focus on calibrating hypothesis likelihood with translation quality by directly optimizing Pearson