Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics
Yilin Zhang Wenda Xu Zhongtao Liu Tetsuji Nakagawa Markus Freitag · 2025
Read Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics by Yilin Zhang Wenda Xu Zhongtao Liu Tetsuji Nakagawa Markus Freitag (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22028.txt.
Introduction
Yilin Zhang, Wenda Xu, Zhongtao Liu, Tetsuji Nakagawa, and Markus Freitag audit the judges used to grade machine translation when no reference translation exists. Quality estimation scores guide data filtering and candidate ranking, so a hidden bias inside the judge quietly reshapes everything downstream.
Across ten language pairs, the team found two length effects. The metrics steadily predict more errors as translations grow longer, even for flawless texts. And when rival translations are equally good, learned metrics tend to prefer the shorter one, while language-model judges range from neutral to favoring longer output. Tracing one leading metric's behavior exposed a built-in penalty that grows with length. The findings warn that pipelines trusting these scores may be optimizing for brevity, not quality.
Catalog record
| CREATOR | Yilin Zhang Wenda Xu Zhongtao Liu Tetsuji Nakagawa Markus Freitag |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22028
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics Yilin Zhang Wenda Xu Zhongtao Liu Tetsuji Nakagawa Markus Freitag Abstract Quality Estimation (QE) metrics are vital in machine translation for reference-free evaluation and as a reward signal in tasks like reinforcement learning. However, the prevalence and impact of length bias in QE have been underexplored. Through a systematic study of top-performing regression-based and LLM-as-a-Judge QE metrics across 10 diverse language pairs, we reveal two critical length biases: First, QE metrics consistently over-predict errors with increasing translation length, even for high-quality, error-free texts. Second, they exhibit a preference for shorter translations when multiple candidates are available for the same source text. These inherent length biases risk unfairly penalizing longer, correct translations and can lead to sub-optimal decision-making in applications such as QE reranking and QE guided reinforcement learning. To mitigate this, we propose two strategies: (a) applying length normalization during model training, and (b) incorporating reference texts during evaluation. Both approaches were found to effectively reduce the identified length bias. Figure 1: QE metrics exhibit a length-related penalty when evaluating long-form translations, as metric scores decrease when translations become longer despite being error-free. As shown, concatenating consecutive gold-standard segments from the same document leads to progressively lower predicted scores, in contrast to human evaluations. 1 Introduction Quality Estimation (QE) metric is a statistical score that predicts the quality of machine-translated text without relying on a reference translation, making it particularly useful when human-produced references are unavailable. Broadly, existing QE methods fall into two categories: regression-based models, such as MetricX ( Juraska et al. 2024 ) and COMET ( Rei et al. 2020 ) , which learn to assess the hypothesis directly from annotated data, and LLM-as-a-Judge approaches, such as InstructScore ( Xu et al. 2023 ) and AutoMQM ( Fernandes et al. 2023 ) , which leverage large language models to generate error ratings that approximate human judgments. Length bias refers to the systematic tendency of evaluation models to prefer or penalize outputs depending on their length rather than their true quality. Prior research has documented such effects in related evaluation settings, including LLM-as-a-Judge frameworks ( Domhan & Zhu 2025 ) and reward models used in reinforcement learning ( Zhao et al. 2025 ) . However, the extent to which QE metrics suffer from length bias remains largely underexplored. This gap is consequential, as length bias can distort system comparisons, over- or under-value longer translations, and ultimately misguide both model development and data curation. In this study, we design two complementary experiments to systematically investigate the presence of length bias in QE metrics. The first experiment evaluates length invariance by concatenating high-quality, error-free segments into progressively longer translations, allowing us to assess whether metrics remain stable as input length increases. The second experiment directly compares alternative hypotheses generated from the same source, enabling us to measure whether metrics exhibit a consistent preference toward shorter or longer outputs. Our analyses reveal two concerning patterns: QE metrics tend to assign spurious errors as translations grow longer, even when no actual errors are present, and they disproportionately favor shorter hypotheses when multiple candidates exist. These findings highlight a fundamental vulnerability—QE metrics risk unfairly penalizing correct but longer translations, which can in turn misguide downstream applications such as system reranking and reinforcement learning pipelines that depend on reliable QE signals. To address the shortcomings, we propose and evaluate two mitigation strategies aimed at reducing length bias in QE metrics. The first approach introduces length normalization during model training, encouraging regression-based QE models to disentangle translation quality from sequence length and thereby improving their robustness. The second approach supplements QE evaluation with reference-based metrics to provide an external standard that counteracts length bias toward shorter outputs. Our experiments show that both strategies are effective in reducing length sensitivity and yield more reliable and equitable evaluations across translations of varying lengths. 2 Preliminaries Definition of Length Bias. Length bias describes a systematic preference for certain output lengths irrespective of true quality. Formally, we evaluate the length bias of QE models using the statistical bias definition. This bias is characterized by the disparity between a predicted quality score and a true quality score, as follows: \textrm{Bias}(\hat{\theta})=\mathbb{E}[\hat{\theta}]-\theta (1) where \mathbb{E}[\hat{\theta}] is an expected quality prediction produced by a QE model and \theta denotes the true translation quality. Ideally, \theta should be derived from human assessments following annotation guidelines such as Multidimensional Quality Metrics (MQM) ( Freitag et al. 2021 ) , and the QE model’s quality prediction is expected to precisely follow the same framework to ensure consistency between \mathbb{E}[\hat{\theta}] and \theta . In our study, all instances within a comparison group are constructed to have the same true quality value \theta . A length-biased QE model, however, would yield consistently different predictions \mathbb{E}[\hat{\theta}] across lengths, which manifests as a measurable \textrm{Bias}(\hat{\theta}) . 3 Experiments Task Definition. A QE model, M_{\text{QE}} , assesses the quality of a system’s hypothesis h given a source text x without relying on human references, assigning a score based on predefined criteria. In our experimental setup, we investigate the phenomenon of length bias by comparing the scores of groups of source–hypothesis pairs (x,h) with the same human-rated translation quality, and the instances in each group are formed by either varying the sequence length of error-free translations or selecting alternative hypotheses of differing lengths for the same source. 3.1 Experiment Setup Dataset. We use the WMT24++ data ( Deutsch et al. 2025 ) for our experiments. WMT24++ contains the human translation and post-edit data for 55 en \rightarrow xx language pairs. We focus on a subset of 10 most common languages, including German, Chinese, Spanish, French, Japanese, Korean, Hindi, Arabic, Russian and Portuguese. One important assumption we make is that the post-edit translations from the WMT24++ dataset are error-free, as the translations are verified by human experts and are of high quality. This ensures that any observed trend in changes of metric scores is not attributable to potential errors in the original data. Quality Estimation Metrics. We consider four popular QE metrics spanning two major evaluation frameworks: (i) two regression-based metrics, MetricX-24 QE ( Juraska et al. 2024 ) and CometKiwi ( Rei et al. 2022 ) , and (ii) two LLM-as-a-Judge variants, AutoMQM fine-tuned on Gemini-2.0 ( Fernandes et al. 2023 ; Finkelstein et al. 2024 ) and directly prompted Gemini-2.5-pro model ( Gemini Team 2025 ) , both of which are used to perform MQM-style evaluation. Hypotheses. We investigate two forms of length bias in this study: (i) bias with respect to source text length and (ii) bias with respect to translation length. For (i), we construct multiple source–translation pairs of equivalent quality to test whether the metrics show systematic preferences related to source text length. For (ii), we compare pairs of translations of equal quality generated from the same source, categorize them by relative length, and analyze whether the metrics display a consistent preference for a particular category. 3.2 Bias in Source Text Length Experiment Setup. To examine bias in source text length, we construct a dataset from the WMT24++ data as follows: for each language pair, we collect the first 5 segments from each document in the data, denoted as s_{1}