Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese

Yikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu · 2025

Read Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese by Yikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12260.txt.

Introduction

Translated text carries a faint accent called translationese: word choices and rhythms that native writing rarely makes. Older tools sort texts into translated or not, a yes-or-no stamp. A nine-author team argues in 2025 that the accent comes in degrees. Their translationese-index scores a text by comparing two small language models, one tuned on originals and one on translations. The ratio of how likely each finds the text becomes the grade.

Trained on only a few thousand synthetic pairs, the index matches human ratings of translatedness. It carries across genres, authors, and language pairs it never saw. It also barely overlaps with standard translation quality scores, so the authors offer it as a fresh yardstick for machine translation research.

Catalog record

CREATORYikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Translationese-index: Using Likelihood Ratios for Graded and Generalizable Measurement of Translationese Yikang Liu, Wanyang Zhang, Yiming Wang, Jialong Tang, Pei Zhang, Baosong Yang, Fei Huang, Rui Wang, Hai Hu Abstract In this paper, we propose the first quantitative measure for translationese—the translationese-index (T-index) for graded and generalizable measurement of translationese, computed from the likelihood ratios of two contrastively fine-tuned language models (LMs). We use a synthesized dataset and a dataset with translations in the wild to evaluate T-index’s generalizability in cross-domain settings and its validity against human judgments. Our results show that T-index is both robust and efficient. T-index scored by two 0.5B LMs fine-tuned on only 1-5k pairs of synthetic data can well capture translationese in the wild. We find that the relative differences in T-indices between translations can well predict pairwise translationese annotations obtained from human annotators; and the absolute values of T-indices correlate well with human ratings of degrees of translationese (Pearson’s r = 0.568 r=0.568 ). Additionally, the correlation between T-index and existing machine translation (MT) quality estimation (QE) metrics such as BLEU and COMET is low, suggesting that T-index is not covered by these metrics and can serve as a complementary metric in MT QE. https://github.com/yikang0131/TranslationeseIndex 1 Introduction Figure 1: Illustration of the pipeline for translationese measuring using likelihood ratios of a pair of LMs. Translationese refers to linguistic properties introduced in the translation process that are different from those of texts originally written in that language ( Gellerstam 1986 ) . While such properties are not inherently undesirable, they often lead to unnatural and non-native-like language that differs from idiomatic and authentic texts. Many attempts have been made to study translationese in the past decades. One approach trains binary classifiers between translated and original texts to uncover the features of translationese ( Baroni and Bernardini 2005 ; Koppel and Ordan 2011 ; Volansky et al. 2015 ; Hu and Kübler 2021 ; Pylypenko et al. 2021 ) ; another relies on the use of the distributional statistics of the features concerning translationese to evaluate the generated texts ( Freitag et al. 2022 ; Guo et al. 2024 ; Li et al. 2025 ; Flamich et al. 2025 ) . Despite these efforts, so far, we do not yet have a quantitative measure for translationese to choose the more authentic and natural translation, out of many equally accurate translations, which is especially important when existing MT QE metrics focus more on the accuracy and become less reliable in evaluating MTs of higher quality ( Agrawal et al. 2024 ; Kocmi et al. 2024 ) . Quantifying the stylistic features at the sample level will facilitate more holistic MT QE. Furthermore, there is a need to isolate translationese from genre-specific features of a text. Specifically, previous studies on translationese are often situated in a binary text-classification paradigm between translated and non-translated text ( Baroni and Bernardini 2005 ; Volansky et al. 2015 ; Pylypenko et al. 2021 ) . Classifiers based on hand-crafted linguistic features rely upon tools such as parsers, and deep-learning-based classifiers might learn features that are specific to the training data but irrelevant to translationese ( Amponsah-Kaakyire et al. 2022 ; Borah et al. 2023 ) . These issues might hinder the generalization of the translationese detection. Against this backdrop, we propose translationese index (T-index), the first method for graded and generalizable measurement of translationese, based on likelihood ratios of two contrastively fine-tuned LLMs, a low-translationese model and a high-translationese model. The method contrasts the distances between the sample and the training distribution of each model, estimating the level of translationese. We have several major findings: (1) Our proposed T-index is supported by/highly correlated with both pointwise and pairwise evaluations of translationese by human annotators. (2) The proposed method is generalizable to multiple genres and authors, even when the backbone LMs are trained on a single domain of only 1k samples, showing robustness and efficiency. (3) T-index has very low correlation with existing MT QE metrics, suggesting that it is a novel aspect not yet covered by existing metrics, and offers a new perspective for MT QE. The paper is organized as follows. We first introduce T-index in § 2 . Next, we describe the synthetic benchmark used for translationese measurement in § 3 . In § 4 , we evaluate the cross-domain generalization of T-index and several baselines on the synthetic benchmark. Finally, we evaluate T-index against human annotations for MTs in the wild in § 5 . 2 T-index: using likelihood ratios to measure translationese Problem formulation. Unlike previous research which was framed as a binary text-classification task between translated and non-translated text, we attempt to provide a continuous measure of translationese. For a source text x and its translation y , we want to estimate the degree of translationese in y . The goal is to find a proper scoring function f(x,y,\theta) , parameterized by \theta . For each sample (x,y) , f will yield a score that positively correlates with the degree of translationese. As reported in the classification between original and translated texts, the supervised classifiers might learn confounding factors that are not necessarily related to translationese, such as the topic of the texts ( Amponsah-Kaakyire et al. 2022 ) . Such features might hinder the cross-domain generalization. Taking this into account, we set two confounding variables: (1) genre and (2) author , which means that the test samples might be sourced from different genres or translated by different authors. The automated translationese measurement is expected to tease out the translationese features from the stylistic variance brought by the genre and author. Suppose that we have a set of authors \mathcal{A}=\{a_{1},a_{2},\ldots,a_{n}\} , a set of genres \mathcal{G}=\{g_{1},g_{2},\ldots,g_{m}\} , we can denote a dataset as \mathcal{D}_{g_{i},a_{i}} , where the source text is sampled from the genre g_{i} , and the translation is produced by the author a_{i} . \mathcal{D} is a paired dataset, each sample containing a translation with a higher degree of translationese ( \tilde{y} ) and a translation with a lower degree of translationese ( y ). The goal can be formalized as follows: \max_{f}\sum_{(x,y,\tilde{y})\in\mathcal{D}}\mathbb{I}\left[f(x,\tilde{y},\theta)>f(x,y,\theta)\right]. Likelihood ratios as translationese index. Inspired by the success of Likelihood Ratios (LLR) in OOD detection ( Ren et al. 2019 ) , we propose to use LLR to measure translationese. Assume that the translation y can be decomposed into three independent parts \{\mathbf{y}_{g},\mathbf{y}_{a},\mathbf{y}_{t}\} as genre component, author component, and translationese component, resulting \log\mathrm{P}_{\theta}(y\mid x)=\log\left[P_{\theta}(\mathbf{y}_{g})P_{\theta}(\mathbf{y}_{a})P_{\theta}(\mathbf{y}_{t})\right] . Given a paired dataset \mathcal{D} , we can contrastively fine-tune two scoring models \theta (on low-translationese samples) and \tilde{\theta} (on high-translationese samples) (see Figure 1 ). T-index can be formalized as: \displaystyle\mathrm{T\mbox{-}index}(y\mid x) \displaystyle=\log\frac{\mathrm{P}_{\tilde{\theta}}(\mathbf{y}_{g})\mathrm{P}_{\tilde{\theta}}(\mathbf{y}_{a})\mathrm{P}_{\tilde{\theta}}(\mathbf{y}_{t})}{\mathrm{P}_{\theta}(\mathbf{y}_{g})\mathrm{P}_{\theta}(\mathbf{y}_{a})\mathrm{P}_{\theta}(\mathbf{y}_{t})} \displaystyle\approx\log\mathrm{P}_{\tilde{\theta}}(\mathbf{y}_{t})-\log\mathrm{P}_{\theta}(\mathbf{y}_{t}). Intuitively, if a translation y is more likely to be high-translationese, the likelihood of y given by the low-translationese model \theta should be lower than that given by the high-translationese model \tilde{\theta} . Since the other two components, genre and author, are shared between the two scoring models, we expect them to be canceled out. We also expect LLR to be robust in cross-domain generalization, because the shift in genre and author in testing samples should be captured by both models. We provide empirical confirmation of these assumptions in Appendix A

Read the complete text