Automatically Generating Chinese Homophone Words to Probe Machine Translation Estimation Systems

Shenbin Qian, Constantin Orăsan, Diptesh Kanojia and Félix do Carmo · 2025

Read Automatically Generating Chinese Homophone Words to Probe Machine Translation Estimation Systems by Shenbin Qian, Constantin Orăsan, Diptesh Kanojia and Félix do Carmo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16158.txt.

Introduction

Shenbin Qian and three co-authors craft tricky Chinese test words to probe whether translation quality checkers really protect emotional tone. User-generated content often carries feelings that a translation should preserve. Emotion-focused evaluation models exist, but their robustness had not been stress-tested.

The method generates emotion-related homophones, words that sound alike but differ in writing and meaning, guided by self-information from information theory. These homophones were observed to trigger emotion-losing translation errors. Human judges rated the generated words, and the new method matched their judgments better than an existing generator. The team then used the words, with manual translations, to perturb texts and expose weaknesses in evaluation models.

Catalog record

CREATORShenbin Qian, Constantin Orăsan, Diptesh Kanojia and Félix do Carmo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16158

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatically Generating Chinese Homophone Words to Probe Machine Translation Estimation Systems Shenbin Qian, Constantin Orăsan, Diptesh Kanojia and Félix do Carmo Abstract Evaluating machine translation (MT) of user-generated content (UGC) involves unique challenges such as checking whether the nuance of emotions from the source are preserved in the target text. Recent studies have proposed emotion-related datasets, frameworks and models to automatically evaluate MT quality of Chinese UGC, without relying on reference translations. However, whether these models are robust to the challenge of preserving emotional nuances has been left largely unexplored. To address this gap, we introduce a novel method inspired by information theory which generates challenging Chinese homophone words related to emotions, by leveraging the concept of self-information . Our approach generates homophones that were observed to cause translation errors in emotion preservation, and exposes vulnerabilities in MT systems and their evaluation methods when tackling emotional UGC. We evaluate the efficacy of our method using human evaluation for the quality of these generated homophones, and compare it with an existing one, showing that our method achieves higher correlation with human judgments. The generated Chinese homophones, along with their manual translations, are utilized to generate perturbations and to probe the robustness of existing quality evaluation models, including models trained using multi-task learning, fine-tuned variants of multilingual language models, as well as large language models (LLMs). Our results indicate that LLMs with larger size exhibit higher stability and robustness to such perturbations. We release 1 1 1 https://github.com/surrey-nlp/homo_gen our data and code for reproducibility and further research. 1 Introduction Machine translation (MT) of Chinese-English news articles has been claimed to achieve human parity in recent years ( Hassan et al., 2018 ) . However, research on machine translation of user-generated content (UGC) like tweets has revealed additional challenges including problems with handling slang, emotion, and literary devices like sarcasm and euphemisms ( Saadany et al., 2023 ) , as shown in the example translated by ChatGPT 2 2 2 Using https://chatgpt.com/ in December 2024. and Google Translate in Figure 1 . Evaluating MT quality of such texts has become a challenging and urgent task for the improvement their translation quality ( Qian et al., 2024c ) . Figure 1: An example of the challenges for translating Chinese UGC Traditional ways of evaluating MT quality involve metrics such as BLEU ( Papineni et al., 2002 ) , BLEURT ( Sellam et al., 2020 ) or BERTScore ( Zhang et al., 2019 ) to compare the MT output with one or several reference translations. When references are unavailable, quality estimation (QE) methods are often used to predict scores which approximate human evaluation ( Specia et al., 2018 ) . One approach for QE is fine-tuning multilingual pre-trained language models (PTLMs) using human evaluation scores. Frameworks like Multi-dimensional Quality Metrics (MQM) ( Lommel et al., 2014 ) , an error-based evaluation scheme, are commonly employed to obtain the human evaluation scores for this purpose. For machine translation of UGC, Qian et al. (2023) recruited professional translators to evaluate translations of a Chinese UGC dataset using Google Translate, based on an MQM-adapted framework. They found that homophone slang words used by netizens are the most common cause of errors in the translation of emotions. They proposed different types of QE models based on fine-tuning, multi-task learning (MTL) and large language models (LLMs) for automatic evaluation ( Qian et al., 2024c ; Qian et al., 2024b ) and claimed that their models achieved state-of-the-art performance in evaluating MT quality of UGC. In this paper, we investigate whether their models are robust enough to cope with newly generated homophones or human-improved translations. In this regard, we propose a method to automatically generate Chinese homophone words to probe the robustness of these QE systems towards new homophone words and human-improved translations. Our contributions can be summarized as follows: • We leverage self-information in information theory for the generation of Chinese homophones that can be used to replace the original word to create new slang, as a novel method. • We compare the proposed method with an existing one using percentile score . We evaluate the two methods based on human evaluation and show that our approach achieves a higher correlation with it. • We utilize generated homophone words and human-improved translations as perturbed examples to probe existing QE models. Our analysis reveals that larger LLMs exhibit greater stability and robustness to our perturbations. The rest of the paper is organized as follows: Section 2 reviews related work on quality evaluation of UGC and Chinese homophone words. Section 3 introduces the main dataset used in this study. Section 4 details the existing generation approach, our proposed method, and the human evaluation and perturbation methods. Section 5 presents and discusses the results of these evaluations. Section 6 concludes the study and outlines future directions, while Section 7 addresses limitations and ethical considerations. 2 Related Work Section 2.1 provides an overview of related work on the evaluation of UGC translation, and Section 2.2 explores studies focused on Chinese UGC and the generation of homophones. 2.1 Evaluation of UGC Translation Despite the tremendous improvement of translation quality since the use of neural machine translation, MT systems still struggle when translating emotion-loaded UGC such as tweets. Saadany et al. (2023) analyzed machine translation of tweets for 6 language pairs and found that hashtags, slang, and non-standard orthography are the most prominent causes of translation errors. Different from the language pairs covered by Saadany et al. (2023) , Qian et al. (2023) analyzed the English translation of Chinese microblog texts. They found that about 50% of their data have translation errors in emotion preservation and about 41% are major and critical errors. Among the causes of errors, emotion-carrying slang that contains homophones is the most frequent cause. To take errors in emotion into consideration during evaluation, Saadany et al. (2021) proposed a sentiment-aware measure for evaluating sentiment transfer by MT systems. Using human evaluation data based on MQM, Qian et al. (2024c) ; Qian et al. (2024b) trained and proposed a series of QE models that can automatically assess MT quality in terms of emotion preservation. They fine-tuned and continued fine-tuned multilingual PTLMs based on TransQuest ( Ranasinghe et al., 2020 ) and COMET ( Rei et al., 2020 ; Stewart et al., 2020 ; Rei et al., 2022 ) , two commonly-used QE frameworks. They also utilized the Nash ( Navon et al., 2022 ) and Aligned ( Senushkin et al., 2023 ) MTL losses to train models that can perform sentence- and word-level QE concurrently. With the recent advancement of LLMs, Qian et al. (2024b) proposed to prompt and parameter-efficiently fine-tune LLMs for quality estimation of emotion-loaded UGC. They claimed to achieve state-of-the-art results using LLMs for evaluation. However, none of these papers answered the question: Are these models robust to new homophone slang words? For this purpose, we propose a method to automatically generate homophone words to test the robustness of their systems. 2.2 Chinese Homophone Words There have been extensive debates about what a word is in Chinese in both natural language processing and linguistic studies, as Chinese does not have a clear delimiter for word boundaries like spaces in English.

Read the complete text