MAATS: A Multi-Agent Automated Translation System Based on MQM Evaluation

2025

Read MAATS: A Multi-Agent Automated Translation System Based on MQM Evaluation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14848.txt.

Introduction

George Wang, Jiaqian Hu, and Safinah Ali staff a translation agency with AI agents. Their 2025 paper presents MAATS, a system where specialist agents each police one quality dimension from a standard metrics framework: accuracy, fluency, style, terminology. A synthesis agent then merges their notes to revise the draft.

Unlike a single model correcting itself, the divided labor catches layered errors, including omissions one perspective spots and another misses. Across language pairs and base models, MAATS beat zero-shot and single-agent baselines in automatic scores and human judgment, with the largest gains in meaning accuracy, locale fit, and distant language pairs. The authors frame the design as a bridge between black-box translation and human editorial workflow.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14848

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MAATS: A Multi-Agent Automated Translation System Based on MQM Evaluation Abstract We present MAATS , a M ulti A gent A utomated T ranslation S ystem that leverages the Multidimensional Quality Metrics (MQM) framework as a fine-grained signal for error detection and refinement. MAATS employs multiple specialized AI agents, each focused on a distinct MQM category (e.g., Accuracy, Fluency, Style, Terminology), followed by a synthesis agent that integrates the annotations to iteratively refine translations. This design contrasts with conventional single-agent methods that rely on self-correction. Evaluated across diverse language pairs and Large Language Models (LLMs), MAATS outperforms zero-shot and single-agent baselines with statistically significant gains in both automatic metrics and human assessments. It excels particularly in semantic accuracy, locale adaptation, and linguistically distant language pairs. Qualitative analysis highlights its strengths in multi-layered error diagnosis, omission detection across perspectives, and context-aware refinement. By aligning modular agent roles with interpretable MQM dimensions, MAATS narrows the gap between black-box LLMs and human translation workflows, shifting focus from surface fluency to deeper semantic and contextual fidelity. Code and data: https://github.com/maats0519/maats_mqm . George Wang 1 Jiaqian Hu 2 Safinah Ali 1 1 New York University, New York, NY, USA 2 Middlebury Institute of International Studies, Monterey, CA, USA {xw3617, sa9140}@nyu.edu , [email protected] 1 Introduction Figure 1: Comparison of Zeor-Shot, Single-Agent and MAATS Pipelines. MAATS assigns MQM dimensions to specialized agents, whose outputs are synthesized by a central agent Lommel et al. (2024) . Figure 2: Translation Metrics Comparison for GPT-4o Across Approaches and Language Pairs. Full-scale versions see Table 16 . While both multi-agent systems and machine translation have seen rapid progress Manakhimova et al. (2023) ; Peng et al. (2023) ; Jiao et al. (2023) , the use of explicitly specialized agents assigned to distinct evaluation tasks remains underexplored. Despite advances that enable LLMs to rival dedicated MT systems, even top-performing models still produce subtle mistranslations, omissions, stylistic errors, and terminology inconsistencies Freitag et al. (2021a) ; Jiao et al. (2023) ; Yan et al. (2024) . Researchers are drawing inspiration from professional human translation workflows, which typically involve a multi-stage process with collaboration between translators and editors Yan et al. (2014) ; Qian and Kong (2024) ; Wu et al. (2024) . This insight has led to refinement approaches for LLMs, including single-agent methods where a model attempts to critique and correct its own output Feng et al. (2024) . Although such self-refinement can reduce errors, a single model can struggle to identify all issues due to potential biases or a lack of diverse expertise Xu et al. (2024) ; Kamoi et al. (2024) , which may have implications on errors in machine translation. This paper introduces MAATS (Multi-Agent Automated Translation System), a novel framework designed explicitly to model a collaborative annotation + refinement process using specialized LLM-based agents. It uses the MQM framework as a shared language for annotating translation errors across multiple dimensions like accuracy, fluency, and style Lommel et al. (2024) . All annotations are synthesized by an Editor agent to produce a refined translation. Our findings are: 1) MAATS consistently outperforms two baselines: zero-shot and single-agent approaches by correcting more critical errors with higher scores on both standard and neural metrics 2) Statistical analysis confirms that MAATS shows significant neural metric gains for linguistically distant pairs and weaker base LLMs. 3) Compared to human MQM annotations, MAATS increases true positives and reduces false negatives over both baselines. 4) Human evaluation shows professional translators ranked MAATS highest over two baselines across models. 5) Qualitative analysis with case studies shows MAATS performs deep error analysis and context-aware translation refinement. 2 Method 2.1 MAATS Design Unlike traditional approaches that combine translation, evaluation, and refinement in one linear step Feng et al. (2024) , MAATS introduces a modular framework where specialized LLM agents collaboratively enhance translation quality through targeted error detection and correction. Shown in Figure 1, the process begins with a Translator Agent, which generates the initial translation using base LLMs. This output is then evaluated by a set of MQM Evaluator Agents, each aligned with a specific MQM category: Accuracy, Fluency, Locale Convention, Audience Appropriateness, Style, Terminology, and Design & Markup Lommel et al. (2024) . These agents independently annotate translation errors within their domain with critical, major, minor severity levels. All annotations are passed to a centralized Editor Agent, which synthesizes and prioritizes the suggested corrections. The Editor prioritizes resolving critical issues, then addresses less severe ones by severity. This modular design avoids iterative feedback loops and ensures efficient integration of evaluations without redundancy or annotation conflicts. Overall, the MAATS architecture delivers higher translation quality and interpretability by simulating the collaborative workflows of human translator-editor teams. 2.2 Prompts and Models MAATS employs distinct prompts for the Translator, specialized MQM Evaluators (See Appendix Accuracy Annotation Prompt ), and Editor agents (Appendix I ). These prompts were designed using a few-shot approach based on real examples adapted from Unbabel’s Typology 3.0. 1 1 1 https://help.unbabel.com/hc/en-us/articles/6444304419479-Annotation-Guidelines-Typology-3-0 For the Single-Agent baseline, a self-refinement prompt is used (Appendix G ). Experiments were conducted using three state-of-the-art large LLMs as base models: Claude-3-haiku, Gemini-2.0-flash, and GPT-4o Anthropic (2024) ; DeepMind (2025) ; OpenAI (2024) . These models served as the base for all three approaches: zero-shot, single-agent, and MAATS. Implementation details for both MAATS and the Single-Agent baseline are provided in Appendix A . Figure 3: MAATS vs. Single-Agent Total Annotation Counts. GPT found the most errors, followed by Claude and Gemini 2.3 Tasks and Evaluation To evaluate the MAATS system, translation tasks were conducted bidirectionally between English and six target languages: German (DE), Hebrew (HE), Japanese (JA), Russian (RU), Chinese (ZH), and Arabic (AR) from WMT 2023 2 2 2 WMT 2023: https://www.statmt.org/wmt23/ and WMT 2024 3 3 3 WMT 2024: https://www.statmt.org/wmt24/ . To comprehensively evaluate the effectiveness of the MAATS system, we conducted six distinct evaluations, each addressing a different aspect of translation quality. Automated Metric Comparison. MAATS was benchmarked against two baselines using standard evaluation metrics, including BLEU, METEOR, BLEURT, and COMET Papineni et al. (2002) ; Banerjee and Lavie (2005) ; Sellam et al. (2020) ; Rei et al. (2020) . Statistical Testing. ANOVA and paired bootstrap resampling were applied to assess the reliability of observed improvements, confirming that MAATS’s gains were statistically significant. Human-Annotated MQM Evaluation. We validated MAATS’s error detection performance by comparing its outputs against human-labeled MQM reference data using confusion matrices Freitag et al. (2021b) . Human Preference Ranking. Professional bilingual translators independently ranked translations from MAATS, single-agent, and zero-shot systems. Qualitative Case Studies. Three detailed case studies highlighted MAATS’s strengths in handling cultural references, multi-perspective error detection, and context-aware refinement. 3 Results MAATS vs. Baselines Annotation. We evaluated the scalability and sensitivity in error annotation of MAATS across different LLMs. MAATS identified 41,547 translation issues across all language pairs, Figure 4: Chinese to English Annotation Analysis Using MAATS and Single Agent Figure 5: Human Evaluation Results: MAATS vs. Single-Agent and Zero-Shot in EN→ZH Translation Figure 6: MAATS Corrects Cultural and Contextual Errors in “Book the Fare. By flagging Accuracy, Style, Audience, and Terminology, “预定票价” was revised to the more natural and contextually appropriate “预定机票". compared to 9,217 errors detected by the single-agent approach—an increase about 450% (Fig.

Read the complete text