Evaluating Language Translation Models by Playing Telephone

Syeda Jannatus Saba, Steven Skiena · 2025

Read Evaluating Language Translation Models by Playing Telephone by Syeda Jannatus Saba, Steven Skiena (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19611.txt.

Introduction

Translation quality is now so high that judging it well has become the bottleneck. Syeda Jannatus Saba and Steven Skiena propose a telephone-game test: translate a text back and forth many times and watch it drift. Texts that drift in telling ways reveal where translators stumble. The pairs produced this way become training data for scoring systems, with no human labels needed.

Their scorers beat a popular evaluation system on two tasks. One task grades a translation against a human reference. The other asks which of two translations sits closer in the chain to the source. Because the data builds itself, the method scales to long and literary texts. Better automatic judges could unlock training gains that current scores cannot see.

Catalog record

CREATORSyeda Jannatus Saba, Steven Skiena
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-19611

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Language Translation Models by Playing Telephone Syeda Jannatus Saba, Steven Skiena Abstract Our ability to efficiently and accurately evaluate the quality of machine translation systems has been outrun by the effectiveness of current language models—which limits the potential for further improving these models on more challenging tasks like long-form and literary translation. We propose an unsupervised method to generate training data for translation evaluation over different document lengths and application domains by repeated rounds of translation between source and target languages. We evaluate evaluation systems trained on texts mechanically generated using both model rotation and language translation approaches, demonstrating improved performance over a popular translation evaluation system ( xCOMET ) on two different tasks: (i) scoring the quality of a given translation against a human reference and (ii) selecting which of two translations is generationally closer to an original source document. 1 Introduction Machine translation (MT) has become a fundamental pillar of multilingual communication, enabling rapid and scalable information transfer across languages. However, our ability to efficiently and accurately evaluate the quality of machine translation systems has been outrun by the effectiveness of current language models—which limits the potential for further improving these models on more challenging tasks like long-form and literary translation. Accurate translation evaluation is needed to inform model development, assess training strategies, and help with real-world deployment decisions in commercial applications. Properly evaluating translation systems is a complex task, balancing issues of correctness and readability. Multiple valid translations exist for every given text, and proper assessment requires capturing nuances in semantics, style, and fluency. Standard metrics like BLEU Papineni et al. 2002 rely heavily on n-gram overlap with a single human reference translation, often penalizing semantically correct yet stylistically or structurally divergent translations. Recent neural metrics such as COMET Rei et al. 2020 and xCOMET Guerreiro et al. 2024 use machine learning to train high-quality evaluators from human-annotated reference translations. However, human annotation is costly, and requires specialized expertise to prepare and score gold standard translations, particularly between pairs of low resource languages. These evaluation techniques will not scale as the research frontier in translation advances from single-sentence translation to the greater challenges of technical and literary translation of poetry, full-length narratives, and textbooks. Figure 1: Language rotation setup for iterative translation. Starting from the Czech sentence “Zemětřesení kolem Putina?”, we rotate through Japanese and Pashto. Meaning gradually shifts from a factual query to subjective concern. In this work, we propose a novel unsupervised method to generate training data for translation evaluation over different document lengths and application domains—without requiring any human annotation. Our approach is inspired by the popular children’s game “Telephone”. Here the children playing order themselves in a line, and the first child makes up a message they whisper to the second child. This process repeats until the end of the line, and with each stage in transmission some fidelity is usually lost. Merriment ensues when the last child reveals what they think the initial message was. We leverage this idea to generate training data for language translation. Starting from an initial text, we use existing language translation models through repeated rounds of translation between source and target languages. Just as with the children’s game, each successive translation can be assumed to be more corrupted with respect to the original source than its predecessor, providing a natural measure of translation quality without human annotation. Careful analysis of such translation sequences allows us to produce pseudo-labels that capture both variations in intrinsic sentence difficulty and the dynamics of semantic drift over multiple translation cycles. Our primary contributions include: • Translation evaluation without human annotation : We introduce our “Game of Telephone” approach, providing an unsupervised pipeline to generate unlimited training data from iterative translations scored by existing automatic metrics. This method substantially reduces the barrier to developing robust MT evaluation models, especially for low-resource language scenarios where human annotations are scarce or unavailable. In the reference evaluation task (given a source text in language L_{1} and its reference translation in another language L_{2} , score a second translation into L_{2} ), our telephone-trained model outperformed (0.604) the state-of-the-art xCOMET Guerreiro et al. 2024 in Pearson correlation (0.514) against the human reference standard. • Language rotation vs. model rotation : We systematically explore two variants of our approach: language-rotated translation cycles (varying the pivot languages at each iteration) and model-rotated translation cycles (varying MT models at each iteration). We empirically demonstrate both strategies can capture nuanced translation degradation: both language- and model-rotation trained models obtained higher AUC scores than xCOMET at predicting the earlier generation texts for five different LLMs. • New translation evaluation benchmark : We have released evaluation metrics (both code and data) to assess translator systems for sensitivity to semantic drift as well as sentence-level accuracy/robustness 1 1 1 https://github.com/saba-phoenix/language-translation-evaluation-by-playing-telephone . This serves as a resource for advancing language translation quality assessment methodologies. Our paper is organized as follows. We survey previous work in translation evaluation in Section 2 . Details of our approach to quality scoring as a function of translation round, document difficulty, and relative model fidelity are discussed in Section 3 . We explain how we use this data to train evaluation models in Section 4 , with our experimental results in Section 5 . We conclude with ideas for future research in Section 6 . 2 Previous Work Machine Translation. MT has advanced with large transformers—from the original Transformer Vaswani et al. 2017 to multilingual models like mBART Liu et al. 2020 , M2M-100 Fan et al. 2021 , and NLLB-200 Costa-jussà et al. 2022 . Instruction-tuned LLMs such as GPT-4 OpenAI 2023 , Mixtral Jiang et al. 2024 , and Gemini Team et al. 2023 now handle translation as a byproduct of language understanding. Still, low-resource performance lags Ruder et al. 2021 ; Freitag et al. 2022a , and issues like idioms and domain shift demand better evaluation and adaptation. Figure 2: Model rotation for iterative translation. We perform three rounds of Turkish-English round-trip translations on “Futbolcu Başkan’ın ayağına gelir”. Later iterations reveal compounding semantic drift. Translation Evaluation System. While MT models have advanced rapidly, evaluation metrics have lagged behind. Traditional metrics such as BLEU Papineni et al. 2002 , METEOR Banerjee and Lavie 2005 , and TER Snover et al. 2006 rely on surface-level n-gram overlap, frequently failing to capture semantic adequacy or fluency. This gap has driven the development of learned metrics more aligned with human judgment, such as COMET and BLEURT Sellam et al. 2020 . Benchmarks like WMT Freitag et al. 2021b , MLQE-PE Fomicheva et al. 2020 , and FLORES Team et al. 2022 have become central to evaluating translation quality. COMET and xCOMET. Learned metrics like COMET and its extensions COMETKIWI Rei et al. 2022 , xCOMET have improved MT evaluation by aligning closely with human judgments, incorporating uncertainty estimates Glushkova et al. 2021 and span-level error detection. Yet recent findings ( Agrawal et al. 2023 ) reveal a saturation effect at the upper end of the quality spectrum, where these metrics struggle to distinguish between strong translations. This is particularly limiting when comparing top-tier systems or reranking hypotheses. Synthetic Data Generation. Tuan et al. 2021 create synthetic QE data by

Read the complete text