Large Language Models as Annotators for Machine Translation Quality Estimation
Sidi Wang, Sophie Arnoult, Amir Kamran · 2026
Read Large Language Models as Annotators for Machine Translation Quality Estimation by Sidi Wang, Sophie Arnoult, Amir Kamran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10775.txt.
Introduction
Sidi Wang, Sophie Arnoult, and Amir Kamran use large language models as annotators to train a machine translation quality estimation model, avoiding the high cost of using big models directly. They take segment-level annotations as the key signal, following earlier work. A simplified MQM scheme guides the model. A prompt pattern built around GPT-4o then produces the annotations. Those labels correlate well with human ones. Training COMET on them gives competitive segment-level scores for Chinese-English and English-German.
Catalog record
| CREATOR | Sidi Wang, Sophie Arnoult, Amir Kamran |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24038 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10775
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Language Models as Annotators for Machine Translation Quality Estimation Sidi Wang, Sophie Arnoult, Amir Kamran Abstract Large Language Models (LLMs) have demonstrated excellent performance on Machine Translation Quality Estimation (MTQE), yet their high inference costs make them impractical for direct application. In this work, we propose applying LLMs to generate MQM-style annotations for training a COMET model: following Fernandes et al. 2023 , we reckon that segment-level annotations provide a strong rationale for LLMs and are key to good segment-level QE. We propose a simplified MQM scheme, mostly restricted to top-level categories, to guide LLM selection. We present a systematic approach for the development of a GPT-4o-based prompt, called PPbMQM (Prompt-Pattern-based-MQM). We show that the resulting annotations correlate well with human annotations and that training COMET on them leads to competitive performance on segment-level QE for Chinese-English and English-German. 1 Introduction Machine Translation (MT) Quality Estimation (QE) metrics fine-tuned on human judgement data like CometKIWI ( Rei et al. 2022 ; Rei et al. 2023 ) now perform on-par with reference-based metrics ( Freitag et al. 2021b ; Freitag et al. 2022 ; Freitag et al. 2023 ; Freitag et al. 2024 ; Lavie et al. 2025 ) . Large Language Models (LLMs) like GPT ( Brown et al. 2020 ) and PaLM ( Chowdhery et al. 2023 ; Anil et al. 2023 ) exhibit human-level performance on various benchmark NLP tasks ( Huang and Chang 2023 ) . Kocmi and Federmann 2023b were the first to utilize LLM prompting for QE, prompting GPT models to output Direct-Assessment (DA) scores. Their GPT-4-based metric GEMBA showing state-of-the-art performance at the system level, where QE metrics have to rank (test)set outputs of translation systems, and promising results at the segment level, where scores are assigned to sentence translations. Fernandes et al. 2023 prompt PaLM models to output Multidimensional Quality Metric (MQM) annotations ( Burchardt 2013 ; Lommel et al. 2014 ) , which mark error spans in translations with a type and degree of severity, as shown in Figure 1 , and have been introduced in the WMT evaluation campaigns following Freitag et al. 2021a . Prompting for MQM provides interpretable results and also triggers reasoning in LLMs, and Fernandes et al. 2023 show that their few-shot AUTOMQM prompt has high recall on major errors and improved segment-level predictions with the larger PaLM-2 Unicorn model ( Anil et al. 2023 ) compared to DA-score prompting. Figure 1: Example MQM annotation. Error spans are marked with error type and severity. MQM annotations remain hard to operationalize, and other QE studies have proposed to identify error spans with their severity only and not with the error types ( Perrella et al. 2022 ; Lu et al. 2023 ; Rei et al. 2023 ; Lu et al. 2024 ; Kocmi et al. 2024b ) , as the scores required for training QE learned metrics are obtained by weighing errors based on their severity, but also because it is sufficient for post-editing tasks ( Kocmi and Federmann 2023a ; Zouhar et al. 2024 ) . This also has resulted in the recent replacement of MQM by ESA Kocmi et al. 2024b in the WMT General Machine Translation shared task Kocmi et al. 2024a . Rather than discarding all category information, we simplify the MQM scheme to top-level categories, and adding the Omission subcategory, which allows us to analyse the error types made by different LLMs. Inference with LLMs is slow and costly ( Rei et al. 2023 ) , therefore we propose to use LLMs to generate synthetic MQM-style annotations for finetuning learned metrics . We make the following contributions: • We introduce a few-shot PPbMQM prompt, restricted to the top-level MQM categories for error type and the omission category. • Our analysis on Chinese-English confirms that LLMs are overly critical; we show that prompting along a severity scale as a basis for score computation improves correlation with human judgments. • We apply our prompt to generating training data for COMET-QE 1 1 1 GitHub: https://github.com/Unbabel/COMET for Chinese-English and English-German, showing positive results with Pearson correlation compared to human annotations. 2 Prompt development for MQM 2.1 Experiment design We develop our prompt in four steps: 1) testing LLMs on their knowledge of QE and MQM; 2) identifying issues with a first zero-shot prompt; 3) zero-shot prompting; 4) few-shot prompting. The first three steps are presented in section 2.2 , and few-shot prompting in section 2.3 . The MQM annotation scheme employs a refined hierarchical error typology. To simplify the task for LLMs, we initially restricted ourselves to the top-level MQM categories Accuracy , Fluency , Terminology , Style , and Locale Convention –adding the subcategory Omission for few-shot experiments–and to two severity levels, Major and Minor . Following White et al. 2023 , we apply the persona and the reflection patterns to elicit better informed predictions, and the output automater pattern for structured output. The rationale details can be found in Appendix A.1 . We experimented with four large LLMs, GPT-3.5, GPT-4 Turbo, GPT-4o, and LLaMA 3 70B 2 2 2 We used APIs provided by OpenAI and Replicate. on Chinese-English, selecting the best performing models after each step. Our dataset is from the Chinese-English and English-German Expert-Based Human Evaluations (EbHE-WMT-MT) of the 2022 and 2023 WMT general MT task ( Freitag et al. 2021a ) . 2.2 Zero-shot prompting We first tested the translation knowledge of the four LLMs by asking questions related to QE and MQM, and scoring answers on a scale of 1 to 5. GPT-3.5 performed the poorest, scoring 14 out of 25, compared to 19 or 20 for the other models (see Table 1 ), and was then excluded from further analysis. Question GPT-3.5 GPT-4 Turbo GPT-4o LLaMA 3 1. Can you explain what machine translation quality estimation is in 100 words? 3 4 4 4 2. Could you provide an overview of the Multidimensional Quality Metrics (MQM) annotation scheme in around 100 words? 3 4 4 4 3. What are the core error categories of MQM? 2 3 4 4 4. How can I effectively evaluate the Chinese sentence and its English translation using the MQM annotation scheme? Provide a concise response within 100 words, please. 3 5 5 4 5. How would you, as a language model, annotate the translation of a Chinese source sentence into English using the MQM scheme? Please provide an example. 3 4 3 3 Total score 14 20 20 19 Table 1: Exploring LLMs’ knowledge of QE and MQM. In each API request, we presented the LLMs with one single question. Answers are scored on a scale of 1-5 (very poor-very good) by a single annotator. Secondly, we designed a zero-shot prompt that relies on LLMs’ inherent knowledge of MQM , shown in Figure 2 . [System message] You are a professional Chinese-English translator. [Instruction message] Your task is to identify translation errors from a pair of the source Chinese sentence and the target English sentence. Please identify up to 5 errors and assign an error type for each with a severity label from "major" and "minor", using the MQM annotation scheme. Please provide the output only in json format: 1. error type