Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement
Gabriele Sarti Vilém Zouhar Malvina Nissim Arianna Bisazza · 2025
Read Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement by Gabriele Sarti Vilém Zouhar Malvina Nissim Arianna Bisazza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23183.txt.
Introduction
Gabriele Sarti and three co-authors study how to spot errors in machine translation, word by word, in a 2025 paper. Such checks help human translators fix drafts fast. The best-known checkers lean on pricey large models or on big piles of hand-labeled data.
The authors test cheaper routes that read a translation model's inner signals and its doubt. They scored 14 measures across 12 language pairs. A catch runs through the work: human judges often disagree on the labels, and that spread caps how good any checker can look. Scores built on one judge's labels prove brittle.
Catalog record
| CREATOR | Gabriele Sarti Vilém Zouhar Malvina Nissim Arianna Bisazza |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement Gabriele Sarti Vilém Zouhar Malvina Nissim Arianna Bisazza Abstract Word-level quality estimation (WQE) aims to automatically identify fine-grained error spans in machine-translated outputs and has found many uses, including assisting translators during post-editing. Modern WQE techniques are often expensive, involving prompting of large language models or ad-hoc training on large amounts of human-labeled data. In this work, we investigate efficient alternatives exploiting recent advances in language model interpretability and uncertainty quantification to identify translation errors from the inner workings of translation models. In our evaluation spanning 14 metrics across 12 translation directions, we quantify the impact of human label variation on metric performance by using multiple sets of human labels. Our results highlight the untapped potential of unsupervised metrics, the shortcomings of supervised methods when faced with label uncertainty, and the brittleness of single-annotator evaluation practices. † † footnotetext: † Materials: gsarti/labl/examples/unsup_wqe . 1 Introduction Word-level error spans are widely used in machine translation (MT) evaluation to obtain robust and fine-grained estimates of translation quality ( Lommel et al. 2014 ; Freitag et al. 2021a ; Freitag et al. 2021b ; Kocmi et al. 2024b ) . Due to the cost of manual annotation, word-level quality estimation (WQE) was proposed for assisting in annotating error spans over MT outputs ( Zouhar et al. 2025 ) . Modern WQE approaches generally rely on costly inference with large language models (LLMs) or ad-hoc training with large amounts of human-annotated texts ( Fernandes et al. 2023 ; Kocmi and Federmann 2023 ; Guerreiro et al. 2024 ) , making them impractical for less resourced settings ( Zouhar et al. 2024 ) . To improve the efficiency of MT quality assessment, several works explored the use of signals derived from the internals of neural MT systems ( Fomicheva et al. 2020b ; Fomicheva et al. 2021 ; Leiter et al. 2024 ) , for identifying problems in MT outputs, such as hallucinations ( Guerreiro et al. 2023a ; Guerreiro et al. 2023b ; Dale et al. 2023a ; Dale et al. 2023b ; Himmi et al. 2024 ) . However, previous works focus on sentence-level metrics for overall translation quality, and do not evaluate performance on multiple label sets due to high annotation costs ( Fomicheva et al. 2022 ; Zerva et al. 2024 ) . 1 1 1 Other relevant works are discussed in Appendix A Figure 1: Example of German \rightarrow English translation with two sets of human word-level error span annotations and two examples of continuous and binary WQE metrics. In this work, we conduct a more comprehensive evaluation spanning 10 unsupervised metrics derived from models’ inner representations and predictive distributions to identify translation errors at the word level. We test three open-source multilingual MT models and LLMs of different sizes across 12 translation directions, including typologically diverse languages and challenging textual domains. Importantly, we focus on texts with multiple human annotations to measure the impact of individual annotator preferences on metric performance, setting a “human-level” baseline for the WQE task. We address the following research questions: i) How accurate are unsupervised WQE metrics in detecting MT errors compared to trained metrics and human annotators? ii) Are popular supervised WQE metrics well-calibrated? iii) Are the relative performances of WQE metrics affected by the variability in human error annotations? We conclude with recommendations for improving the evaluation and usage of future WQE systems. DivEMT WMT24 QE4PE Languages en \rightarrow ar,it, nl,tr,uk,vi en \rightarrow ja,zh, hi,cs,ru cs \rightarrow uk en \rightarrow it,nl Errors type Post-edit Annotation Post-edit Label sets 1 1 6 Domains Wiki Multiple Social, Biomed MT Model mBART-50 Aya23 NLLB # Segments 2580 5124 3888 Table 1: Summary of tested datasets. Error spans are obtained from explicit error annotations or post-edited spans. Additional details are available in Appendix B . 2 Data We use datasets containing error annotations or post-edits on the outputs of open-source models to extract unsupervised WQE metrics on real model outputs, avoiding possible confounders. We select the following datasets, summarized in Table 1 : DivEMT ( Sarti et al. 2022 ) contains a single set of post-edits over translations produced by mBART-50 ( Tang et al. 2021 ) for a subset of Wiki texts from the FLORES dataset ( Goyal et al. 2022 ) spanning six typologically diverse target languages ( en \rightarrow ar,it,nl,tr,uk,vi ). We use it for a cross-lingual comparisons over a fixed set of examples. WMT24 ( Kocmi et al. 2024a ) contains error spans on the outputs of the Aya23-35B LLM ( Aryabumi et al. 2024 ) produced for the WMT24 General Translation Shared Task spanning multiple domains across six directions ( en \rightarrow ja,zh,hi,cs,ru and cs \rightarrow uk ). It was selected to extend our evaluation to a state-of-the-art LLM, given the popularity of such systems in MT ( Kocmi et al. 2023 ) . QE4PE ( Sarti et al. 2025 ) contains multiple human professional post-edits over translations produced by the NLLB 3.3B model ( Costa-jussà et al. 2024 ) for en \rightarrow it and en \rightarrow nl on challenging textual domains (social posts and biomedical abstracts). This dataset is used to conduct our evaluation across multiple annotation sets. 3 Evaluated Metrics The following metrics were evaluated using the Inseq library ( Sarti et al. 2023 ; Sarti et al. 2024b ) . Predictive Distribution Metrics. We use the Surprisal of the predicted token t^{*} , as negative log-probablity -\log p(t^{*}_{i}|t_{<i}) , and the Entropy H of the output distribution P_{N} over vocabulary V , -\sum_{i=1}^{|V|}p(t_{i}|t_{<i})\log_{2}p(t_{i}|t_{<i}) , as simple metrics to quantify pointwise and full prediction uncertainty ( Fomicheva et al. 2020b ) . For surprisal, we also compute its expectation ( MCD avg ) and variance ( MCD var ) with n=10 steps of Monte Carlo Dropout (MCD, Gal and Ghahramani 2016 ) to obtain a robust estimate and a measure of epistemic uncertainty in predictions, respectively. 2 2 2 Epistemic uncertainty reflects models’ lack of knowledge rather than data ambiguity. MCD is tested only on encoder-decoder models since Aya layers do not include dropout. Vocabulary Projections. We use the LogitLens (LL, nostalgebraist 2020 ) to extract probability distributions P_{0},\dots,P_{N-1} over V from intermediate activations at every layer l_{0},\dots,l_{N-1} of the decoder. We use the surprisal for the final prediction at every layer ( LL-Surprisal ) to assess the presence of layers with high sensitivity to wrong predictions. Then, we compute the KL divergence between every layer distribution and the final distribution P_{N} , e.g. \text{KL}(P_{N-1}\|P_{N}) , to highlight trends in the shift in predictive probability produced by the application of remaining layers ( LL KL-Div ). Finally, we adapt the approach of Baldock et al. 2021 and use the number of the first layer for which the final prediction corresponds to the top logit as a metric of model confidence, l\;\text{s.t.}\;\arg\max P_{l}=t^{*} and \arg\max P_{i}\neq t^{*}\;\forall i<l ( LL Pred. Depth ). Context mixing. We use the entropy of the distribution of attention weights 3 3 3 For encoder-decoder model, self-attention and cross-attention weights are concatenated and renormalized. over previous context as a simple measure of information locality during inference ( Ferrando et al. 2022 ;