MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation

Parker Riley Daniel Deutsch Mara Finkelstein · 2025

Read MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation by Parker Riley Daniel Deutsch Mara Finkelstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24664.txt.

Introduction

Parker Riley, Daniel Deutsch, Mara Finkelstein, Colten DiIanni, Juraj Juraska, and Markus Freitag test a two step way to grade machine made translations. In the known MQM scheme, raters mark each error span by type and weight. Their twist adds a second pass where a rater edits marks that are already there.

The first marks may come from the same rater, a new rater, or a machine rater. In trials, the second pass found errors missed the first time, and rater acts lined up with the goals of the scheme. The gain is a grade set with less noise as translation tools get strong.

Catalog record

CREATORParker Riley Daniel Deutsch Mara Finkelstein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24664

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation Parker Riley Daniel Deutsch Mara Finkelstein Abstract Human evaluation of machine translation is in an arms race with translation model quality: as our models get better, our evaluation methods need to be improved to ensure that quality gains are not lost in evaluation noise. To this end, we experiment with a two-stage version of the current state-of-the-art translation evaluation paradigm (MQM), which we call MQM re-annotation. In this setup, an MQM annotator reviews and edits a set of pre-existing MQM annotations, that may have come from themselves, another human annotator, or an automatic MQM annotation system. We demonstrate that rater behavior in re-annotation aligns with our goals, and that re-annotation results in higher-quality annotations, mostly due to finding errors that were missed during the first pass. 1 Introduction A critical component of machine translation (MT) model development is evaluating model quality. Despite extensive work on automatic evaluation ( Guerreiro et al. 2024 ; Juraska et al. 2024 ; Freitag et al. 2024 ; Kocmi and Federmann 2023 ; Fernandes et al. 2023 ; Rei et al. 2020 ) , human evaluation remains the gold standard for reliably determining quality. However, there are still issues affecting the quality of human evaluation, such as differences in rater behavior and task difficulty. These can negatively affect the reliability of human evaluation, and as the quality of MT systems improves, there is a risk that this evaluation noise can outweigh actual quality differences between models, leading to incorrect modeling decisions. As a result, human evaluation must also be improved to facilitate comparing ever-higher-quality systems ( Freitag et al. 2021a ; Kocmi et al. 2024a ; Kocmi et al. 2023 ) . The current state-of-the-art human evaluation framework for MT is Multidimensional Quality Metrics (MQM), originally proposed by Lommel et al. 2014 . Freitag et al. 2021a adapted MQM for MT, arguing that the use of fine-grained annotation by expert raters improved the quality of human evaluation. In MQM, expert raters identify error spans within a translation, assigning a severity and hierarchical category to each. Follow-up work has examined other facets of evaluation that can be used to enhance quality, including comparative judgment ( Song et al. 2025 ) , careful rater assignment ( Riley et al. 2024 ) , and rater calibration ( Kocmi et al. 2022 ) . In this work we explore both human-machine collaboration and human-human collaboration, with the goal of increasing quality of MQM annotation for machine translation evaluation. We do this using re-annotation , where a completed MQM annotation of a document is given to an expert human rater who can delete or modify any previously-marked error spans in addition to adding new ones. The prior rater can either be the same human rater, a different human rater, or an automatic system. Figure 1 illustrates these three cases. Figure 1: Illustration of MQM re-annotation. A source document and its translation are annotated in the MQM framework by either a human or an automatic system, and then re-annotated by a human. If the initial annotator was a human, the re-annotator can be either the same person or a different one. 2 Research Questions and Findings In this section, we outline the research questions we seek to answer in this work and summarize the findings. Evidence supporting these findings is presented in Section 5 , with relevant context and background covered in the intervening sections. 2.1 How much do raters change prior annotations? Our first goal is to understand what happens when raters perform MQM re-annotation: how often do they change or delete prior errors, and on average how many new ones do they add? Further, to what extent does this depend on the source of those prior errors (the same rater, a different human rater, or an automatic system)? We find that rater behavior is highly sensitive to the source of the prior errors, and trends align with our expectations: raters perform the fewest overall modifications to their own annotations, more to other humans’ annotations, and even more modifications to automatic annotations. We also find evidence suggesting that a primary benefit of re-annotation is that it allows raters the opportunity to find errors that were missed initially. 2.2 Do raters detect artificial error spans? One potential risk with re-annotation is that raters may overly trust the prior errors and focus solely on adding new ones. To examine this, we injected artificial error spans and measured how often raters modified or deleted them. We found that, on average, raters removed most of the artificial errors, but that a minority of raters accepted these errors as-is with high frequency. 2.3 Does re-annotating human annotations improve their quality? The prior two research questions establish that re-annotation is reasonable , in that re-annotators behave in ways that are consistent with our expectations, but we also wish to understand whether it is beneficial , in that it improves rating quality. Because defining a ground truth is difficult in human evaluation research, we measure whether re-annotation can allow two disjoint groups of human raters to achieve higher cross-group agreement. We find that re-annotating human MQM ratings (either by the same rater or a different human rater) increases their quality. 2.4 Do automatic prior annotations improve human annotation quality? As a cheaper alternative to human-human re-annotation, we examine whether re-annotating automatic ratings is beneficial over standard MQM annotation from scratch. Based on the previous research question, we can use our re-annotated data as the proxy ground truth to measure improvement. We find that providing raters with prior annotations from high-quality LLM-based automatic systems improves rating quality over from-scratch annotation, at no additional human annotation cost. 3 MQM and Re-Annotation In the Multidimensional Quality Metrics (MQM) framework for human evaluation of machine translation (MT), expert annotators identify error spans within translations and assign a hierarchical category (e.g. Fluency/Grammar or Accuracy/Mistranslation) and severity (Major or Minor). Instead of raters providing scalar scores, scores are calculated by applying a weighting scheme based primarily on severity to each error and summing over the segment; see Freitag et al. 2021a for details. All source and target segments for a single system’s translation of a single document are presented to the rater on the same screen. MQM re-annotation functions the same as standard MQM annotation, except that some error spans are already present when raters first load the interface. Raters have the ability to modify any component of a prior error (span start/end, category, severity), delete it entirely, and add new errors. We use the open-source Anthea 1 1 1 https://github.com/google-research/google-research/tree/master/anthea framework to collect MQM ratings, including re-annotations. The rating interface is shown in Figure 2 . Figure 2: Anthea interface showcasing prior error annotations. 4 Experimental Setup 4.1 Data We use translation data from the WMT 2023 Metrics Shared Task ( Freitag et al. 2023 ) in two language pairs: English-German and Chinese-English. For budget reasons, we only use data from the News domain. Information about the dataset size is shown in Table 1 . The data contains 3 initial human MQM annotations per segment, and two automatic annotations from different systems (Section 4.3 ). Each of these annotations was then re-annotated

Read the complete text