A Measure of the System Dependence of Automated Metrics

Pius von Däniken, Jan Milan Deriu, Mark Cieliebak · 2024

Read A Measure of the System Dependence of Automated Metrics by Pius von Däniken, Jan Milan Deriu, Mark Cieliebak (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03152.txt.

Introduction

Pius von Däniken, Jan Milan Deriu, and Mark Cieliebak raise a fairness point about machine translation scores in 2024. Scores from machines have largely replaced costly human grades. They are most often judged by how well they track human grades.

The authors argue that tracking is not enough. A score should also treat all translation systems in a fair and even way. They propose a way to measure how much a score depends on which system made the text. That check can reveal bias that a simple track test would miss. Builders can use it to pick scores that rank systems more justly.

Catalog record

CREATORPius von Däniken, Jan Milan Deriu, Mark Cieliebak
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03152

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Measure of the System Dependence of Automated Metrics Pius von Däniken, Jan Milan Deriu, Mark Cieliebak Abstract Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect. 1 Introduction Recent years have seen significant advances in machine translation (MT), marked notably by the introduction of the transformer architecture ( Vaswani et al., 2017 ) . Current large-scale commercial systems such as GPT ( Brown et al., 2020 ) continue this trend and show promising results ( Kocmi et al., 2023 ; Hendy et al., 2023 ; Wu and Hu, 2023 ) . A critical supplement to these advancements is thorough and reliable evaluation procedures, which are essential not only for measuring overall progress but also for effectively comparing different systems. While evaluation based on human raters is still considered the gold standard, it is expensive and time-intensive. Therefore, considerable efforts have been made to develop automated metrics for assessing translation quality. Notably, the WMT Metrics series of shared tasks are dedicated to this purpose ( Freitag et al., 2023 ; Freitag et al., 2022 ; Freitag et al., 2021 , i.a.) . Automated metrics usually assign a scalar 1 1 1 Other types of ratings exist, in particular preference ratings ( Belz and Kow, 2010 ) . quality rating to a candidate translation based on the source segment and a reference translation. A system-level rating is derived by averaging the segment ratings over a test set. To measure a metric’s usefulness, we usually measure two aspects: its correlation to human judgments on the segment-level (which checks if there is a monotonic function between metric ratings and human ratings) and whether the system-level ratings can reproduce the same ranking as human ratings ( Kocmi et al., 2021 ; von Däniken et al., 2024 ) . In this paper, we argue that this evaluation of metrics is insufficient, as it ignores a central requirement, namely, that it should treat all systems under evaluation equally. As stated more colloquially, a measuring stick should not change length depending on the measured object. However, this is exactly what we observe in current metrics. Figure 1: Average Human Ratings associated with XCOMET scores on Chinese to English ( zh-en ) WMT 23 data. We show scores for all system in aggregate (global) and two individual systems. Consider Figure 1 , which shows the expected human rating for each score of the XCOMET metric (the best metric in the WMT23 metrics task, with a very high segment-level correlation of 0.65 for the zh-en language pair) ( Freitag et al., 2023 ) . That is, for each possible value that XCOMET may assume, we show the expected human rating and the 95% confidence interval (computed using Isotonic Regression and bootstrap sampling; see Sections 2 and 3 for the details). The global curve (blue) shows the average human score for each metric score if computed over all systems under evaluation in the WMT23 dataset (in standard correlation to human judgment evaluation, we only measure whether this curve is monotonic). In contrast, Lan-BridgeMT (best system according to humans) and NLB-Greedy (lowest-rated system according to humans) show the average human score for each metric score when computed on one separate system only. For instance, an XCOMET score of 0.7 corresponds to an average Human-MQM score of -5.2 for Lan-BridgeMT , and Human-MQM score of -10.2 for NLB-Greedy . This leads to the following consequence: For Lan-BridgeMT , higher human scores are associated with lower metric scores than the global curve, which leads to an underestimation of Lan-BridgeMT ’s performance, according to XCOMET . The opposite effect is visible for NLB-Greedy , which is overestimated and, in fact, gains 3 ranks (from 15th to 12th place) when comparing the metric and human ranking (see also Table 1 in Section 3 ). Thus, a metric that exhibits a high global segment-level correlation to human judgments can lead to wrong system-level rankings. This observation leads us to the central claim of this paper: The cause of the discrepancy between the correlation on the segment level and the final system ranking is due to the metric’s dependency of the system under evaluation. The main position of this paper is that when evaluating a novel metric, one ought to measure the dependency on the system under evaluation as well, alongside the correlation to human judgment. In the following, we will formalize this dependency of the relation between human and metric ratings on the system under evaluation and derive a measure for quantifying this effect. 2 Averaging Metric Scores Assume we are given a set of K machine translation systems \pi_{k} to evaluate. A translation system maps an input sentence i\in\mathcal{I} in a fixed source language to an output sentence o\in\mathcal{O} in a fixed target language: \pi_{k}:\mathcal{I}\rightarrow\mathcal{O} . The usual human evaluation scenario involves curating a test set of N inputs \mathcal{T}=\left\{i^{(j)}|1\leq j\leq N\right\}\subset\mathcal{I} for which we collect the output of each system \pi_{k} for each input i\in\mathcal{T} , and then ask human annotators to produce ratings. This results in a set of ratings \left\{(h_{1}^{(j)},\dots,h_{k}^{(j)},\dots,h_{K}^{(j)})|1\leq j\leq N\right\} , where h_{k}^{(j)}\in\mathbb{R} is a scalar rating provided by human annotators measuring the quality of the translation provided by \pi_{k} for input i^{(j)} . We will assume that higher human ratings indicate higher translation quality. In this setting, it is natural to measure the overall quality of system \pi_{k} by the average human rating it achieves \hat{\mu}_{k}^{H}=\frac{1}{N}\sum_{j=1}^{N}h_{k}^{(j)} . This is an estimator of the expected human rating \mu_{k}^{H}=\mathbb{E}[h_{k}] achieved by \pi_{k} for any input in \mathcal{I} , assuming that \mathcal{T} is appropriately chosen. In many cases, we want to replace human raters with an automated scalar metric M:\mathcal{I}\times\mathcal{O}\rightarrow\mathbb{R} , which maps an input and translation to a scalar value. For our test set \mathcal{T} , we can collect all metric ratings \left\{(m_{1}^{(j)},\dots,m_{k}^{(j)},\dots,m_{K}^{(j)})|1\leq j\leq N\right\} , where m_{k}^{(j)}=M(i^{(j)},\pi_{k}(i^{(j)})) , the metric rating for input i^{(j)} and translation by \pi_{k} . In this case, it is common to use the sample average \hat{\mu}_{k}^{M}=\frac{1}{N}\sum_{j=1}^{N}m_{k}^{(j)} to measure the quality of system \pi_{k} , which is an estimator of the expected metric rating \mu_{k}^{M}=\mathbb{E}[m_{k}] achieved by \pi_{k} . The goal of automated evaluation is to use \hat{\mu}_{k}^{M} as a proxy measure for \mu_{k}^{H} , in particular, to rank the systems \pi_{1},\dots,\pi_{K} according to their performance. In the following, we will study the relationship between \mu_{k}^{H} and \mu_{k}^{M} , which is expressed by an unknown function f_{G} that maps from the metric scale to the human scale. There are two requirements to this function: first, that it is monotonic (i.e., that it respects the order of the metric scale), and second, that it does not depend on the system under evaluation \pi_{k} (i.e., that it is the same for all systems) The goal is to find the relation between \mu_{k}^{H} and \mu_{k}^{M} . The idea is to express \mathbb{E}[h_{k}] in terms of an expectation over metric

Read the complete text