Adding Chocolate to Mint : Mitigating Metric Interference in Machine Translation
José Pombal, Nuno M. Guerreiro, Ricardo Rei, André F. T. Martins · 2025
Read Adding Chocolate to Mint : Mitigating Metric Interference in Machine Translation by José Pombal, Nuno M. Guerreiro, Ricardo Rei, André F. T. Martins (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08327.txt.
Introduction
Using the same metric to tune and to judge a translation system can mislead its makers, José Pombal and co-authors show in a 2025 paper. The problem is called metric interference. As metrics grow stronger and more widely used, outputs become a function of the metric itself, and scores lose their link to human judgments of quality.
The team studies two common cases: filtering training data with a metric, and decoding with quality signals. Interference distorted scores at the level of single items, even when a distinct but related metric was used for judging. Their fix, MINTADJUST, aims at more reliable evaluation under interference. On a shared translation test set, it ranked translations and systems better.
Catalog record
| CREATOR | José Pombal, Nuno M. Guerreiro, Ricardo Rei, André F. T. Martins |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08327
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adding Chocolate to Mint : Mitigating Metric Interference in Machine Translation José Pombal, Nuno M. Guerreiro, Ricardo Rei, André F. T. Martins Abstract As automatic metrics become increasingly stronger and widely adopted, the risk of unintentionally “gaming the metric” during model development rises. This issue is caused by metric interference ( Mint ), i.e. , the use of the same or related metrics for both model tuning and evaluation. Mint can misguide practitioners into being overoptimistic about the performance of their systems: as system outputs become a function of the interfering metric, their estimated quality loses correlation with human judgments. In this work, we analyze two common cases of Mint in machine translation-related tasks: filtering of training data, and decoding with quality signals. Importantly, we find that Mint strongly distorts instance-level metric scores, even when metrics are not directly optimized for—questioning the common strategy of leveraging a different, yet related metric for evaluation that is not used for tuning. To address this problem, we propose MintAdjust , a method for more reliable evaluation under Mint . On the WMT24 MT shared task test set, MintAdjust ranks translations and systems more accurately than state-of-the-art-metrics across a majority of language pairs, especially for high-quality systems. Furthermore, MintAdjust outperforms AutoRank , the ensembling method used by the organizers. 1 1 1 We will release a codebase for replicating the results in this work upon publication. 1 Introduction Figure 1 : MintAdjust learns to predict the instance-level scores of an interfering metric, given the scores of other metrics measured in translations produced by models that are not under Mint (in gray ). At test time, for each instance from the model under Mint , MintAdjust receives the scores of other metrics , and produces a new interfering metric score that correlates better with human judgments. Automatic evaluation metrics are evolving beyond their traditional role in model selection, now serving as tools for enhancing the performance of large language models (LLMs) across their entire development pipeline. Applications include filtering high-quality training data, or serving as proxies for human preferences in alignment and decoding ( Xu et al., 2024b ; Zhu et al., 2024 ; Wu et al., 2024 ) . In the field of machine translation (MT), several works have leveraged such methods to claim state-of-the-art results ( Fernandes et al., 2022b ; Xu et al., 2024b ; Rei et al., 2024 ) . In these cases, the same or similar metrics to the one used for optimization are also used downstream for evaluation—we call this phenomenon metric interference ( Mint ). An unintended consequence of Mint is that it may jeopardize evaluation. For example, an interfering metric can lose much of its correlation with human judgements when ranking systems Fernandes et al. (2022b) , or make the optimized model generate absurd outputs that the metric scores highly ( Yan et al., 2023 ) . At worst, authors disregard Mint and use the same metric for optimization and evaluation. At best, as recommended by Kocmi et al. (2024b) , they acknowledge it and perform evaluation with another metric ( Fernandes et al., 2022b ; Rei et al., 2024 ; Kocmi et al., 2024a ) . The latter solution, however, may not work well if metrics correlate strongly among each other, as they can have similar biases. The effects of Mint are not well understood, and existing mitigation methods lack effectiveness. In the first part of our work, we investigate the implications of Mint for evaluation with a series of metrics on two common Mint cases in MT: 1) data filtering, where a metric is used to select high-quality data from a larger pool to more efficiently train a translation model; 2) MBR, a widely-used decoding strategy that uses a metric to select the best translation from a pool of candidates. We find that the effects of Mint are stronger for MBR than for data filtering. Specifically, the instance-level scores of metrics are heavily distorted, to the point where large improvements in an interfering metric become consistently related to deterioration in other metrics. Importantly, similar distortions hold when comparing an interfering metric with human scores, explaining the system-level findings of ( Fernandes et al., 2022b ) . Furthermore, the scores of metrics that correlate with the interfering metric will also be overestimated. This brings into question the common practice of using a single related metric for evaluation. Building on our findings, we propose a method for correcting an interfering metric’s scores by learning its distribution without Mint from a large set of other metrics, as opposed to relying on a single one: MintAdjust (summarized in Figure 1 ). Using WMT24 human-judged translations, we show that, under Mint , MintAdjust ranks systems and individual translations more accurately than the interfering metric and other state-of-the-art metrics on a majority of language pairs. Its performance is particularly strong when comparing high-quality MT systems. Our correction method generalizes across translation models and metrics— i.e. , it can be learned from a given set of models and metrics and applied to a different model. Through our fine-grained analysis and the proposed methods, we lay the groundwork for better evaluation practices in settings where automatic metrics are used for both evaluation and model optimization. Our work has implications not only for MT but for evaluation in general. 2 Metric Interference When used for evaluation, most automatic metrics \mathcal{M} , may be described as functions that produce a judgement \hat{q} given an input x , a model’s output h , and a reference output r : 2 2 2 In translation, some metrics only use x and h ( e.g. , quality estimation), or h and r ( e.g. , lexical metrics) \hat{q}=\mathcal{M}(x,h(x),r). (1) Mint occurs when \mathcal{M} —the interfering metric —influences a model’s output, directly or indirectly, through its judgements. Thus, during evaluation, \mathcal{M} becomes a function of itself: \hat{q}=\mathcal{M}(x,h(x;\mathcal{M^{\prime}}),r) (2) \mathcal{M^{\prime}} and \mathcal{M} can be the same or spuriously correlated metrics—we will show later that Mint also affects the latter. Direct interference occurs when a metric is used to choose among model outputs at test time; for example, in MBR the best candidate out of a pool of model generations is chosen by a metric. Indirect interference happens when the metric influences the model’s outputs by influencing the model’s weights, \theta , through either data or tuning: \hat{q}=\mathcal{M}(x,h(x;\theta(\mathcal{M})),r) . Training data filtering is a case of indirect interference, whereby the metric influences the model’s weights by changing its training distribution. Preference optimization is an example of indirect interference through tuning. We posit the negative effects of Mint on evaluation are stronger for direct interference. To the best of our knowledge, Mint has never been formally defined but its effects have been studied empirically to some extent. For example, LLM judges tend to favor outputs from their own model family ( Panickssery et al., 2024 ; Verga et al., 2024 ) . In MT, this is also the case with LLMs ( Agrawal et al., 2024 ) and more traditional neural metrics: models optimized for Bleurt can produce absurd translations that score highly ( Yan et al., 2023 ) , while metric-based decoding strategies (e.g., MBR) reduce metric correlation with human judgments