Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation
Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, Markus Freitag · 2025
Read Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation by Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, Markus Freitag (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18771.txt.
Introduction
Muhammed Kocyigit and co-authors measured in 2025 how data contamination inflates language model scores in machine translation. Contamination means test items leak into training data, which can make a benchmark lie. The team built a clean train-test split, then added contamination back in controlled stages, sizes, and formats to isolate its effect.
At both one-billion and eight-billion parameter scales, leaking both the source and the translation sharply raised the standard BLEU score. The inflation was about two and a half times larger for the bigger model, reaching up to thirty points in the report. Leaking only one side gave smaller, less steady effects. Readers get hard numbers for a suspicion many evaluators share: bigger models gain more from leaked tests, so clean splits matter more as models grow.
Catalog record
| CREATOR | Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, Markus Freitag |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18771
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch, Jiaming Luo, Colin Cherry, Markus Freitag Abstract Data contamination—the accidental consumption of evaluation examples within the pre-training data—can undermine the validity of evaluation benchmarks. In this paper, we present a rigorous analysis of the effects of contamination on language models at 1 1 B and 8 8 B scales on the machine translation task. Starting from a carefully decontaminated train-test split, we systematically introduce contamination at various stages, scales, and data formats to isolate its effect and measure its impact on performance metrics. Our experiments reveal that contamination with both source and target substantially inflates bleu scores, and this inflation is 2.5 × 2.5\times larger (up to 30 bleu points) for 8 8 B compared to 1 1 B models. In contrast, source-only and target-only contamination generally produce smaller, less consistent over-estimations. Finally, we study how the temporal distribution and frequency of contaminated samples influence performance over-estimation across languages with varying degrees of data resources. 1 Introduction Scaling laws have reshaped our understanding of the data requirements for training language models ( lm ), leading to a rapid expansion in data collection efforts. This expansion has inadvertently increased the probability of evaluation data contamination ( Sainz et al., 2024 ) : fragments or even entire test sets are accidentally consumed within the pre-training data, thereby invalidating the assumption that models are evaluated on unseen data. Although several studies have acknowledged the contamination issue and shown that it can contribute to performance overestimations ( Zhou et al., 2023 ; Jiang et al., 2024 ; Yang et al., 2023 ) , we still lack a large-scale controlled analysis to better characterize the phenomenon. Concretely, prior efforts are commonly limited to smaller scales, both in terms of model and data ( Jiang et al., 2024 ) , treating contamination as a fine-tuning ( Yang et al., 2023 ) or extended pre-training problem ( Zhou et al., 2023 ) , rather than addressing it directly in the larger-scale pre-training setting. Meanwhile, computational and logistical constraints have deterred large-scale, systematic experiments that would clarify how contamination interacts not only with model size, but also with training dynamics, and data composition. In this paper, we present a controlled study to isolate and measure the impact of contamination during pre-training under diverse contamination conditions and two model scales ( 1 and 8 billion parameters). For this purpose, we take machine translation ( mt ) test suites as our case study, covering eight languages, thereby gaining insight into the interaction between contamination and linguistic resource availability. Starting with a multilingual mixture of public pre-training corpora, we first decontaminate our train-test set splits and train a base lm on the training data (Figure 1 ). Then, we systematically introduce mt test examples into pre-training by varying their mode (presenting source and target in isolation or as full, prompted parallel texts), temporal distribution (controlling for when contamination happens), and frequency (controlling for the number of contamination copies). To efficiently explore various contamination conditions without the prohibitive cost of training from scratch, we adopted a branching strategy—resulting in more than 50\% reduction in our training computational budget. Concretely, each experiment branched from a base model checkpoint and continued training on a modified data mixture, which replaces original samples with test contamination instances. Following our branching strategy, we studied a total of 42 contamination conditions for each model scale, by comparatively studying the performance of contaminated and uncontaminated models across contaminated and non-contaminated datasets. Our findings are summarized as follows: Figure 1 : Large-scale contamination analysis setup: We decontaminate our train-test splits and train a baseline model. Then, insert test data into the pre-training data and train a contaminated model branching out from the baseline checkpoint. Finally, we compare the relative performance of the contaminated and the baseline model on contaminated and non-contaminated data. • Contaminating source-target mt pairs inflates performance on those test sets. Contaminating examples with both source and target sides of the test data leads to substantial performance inflation—up to 30 bleu points–for 8 B-parameter models. On the other hand, partial contamination (source-only or target-only) generally yields smaller and less consistent inflations, especially when the improvements are compared to results on non-contaminated test sets (§ 5.1 ). • The temporal distribution of contamination matters. Contaminating datasets at concentrated points causes large performance inflation at the time of contamination—up to 60 bleu points—with their effects diminishing as training continues. On the other hand, uniformly introducing contamination throughout training has the most persistent impact (§ 5.2 ). • The impact of contamination increases with model scale. Larger models exhibit increased sensitivity to even a single copy of contamination. Increasing the frequency of source-target contaminated examples yields higher performance overestimations, while source-only or target-only contamination is less affected by the frequency of contamination (§ 5.3 ). • Contamination requires sufficient language representation to have a measurable effect. In the absence of language representation in the pre-training data, contamination showed no measurable impact on the languages and model scales studied (§ 5.5 ). Once a certain threshold of language representation is achieved, contamination has a larger impact on language-pairs with lower base model performance (§ 5.6 ). 2 Related Work Prior work has studied contamination mainly by detecting it post-hoc or training models with contaminated data to understand its impact on model performance. The former involves examining token probabilities ( Shi et al., 2024 ) or analyzing model preferences for specific orderings ( Oren et al., 2023 ) , while the latter assesses contamination’s impact by intentionally contaminating test sets into llm s’ pre-training mixtures ( Jiang et al., 2024 ; Yang et al., 2023 ; Magar & Schwartz, 2022 ; Zhou et al., 2023 ) . We present a comparative summary of our method and previous work in Table 1 . Several studies have explored the impact of data contamination by extended pre-training or fine-tuning on contaminated data (Table 1 , Training Process), leaving the generalization of their results to contamination in pre-training an open question. Zhou et al. (2023) set up their experiments as extended pre-training, which could overstate contamination’s impact, as models are more susceptible to performance inflation when contamination is introduced later in training ( Magar & Schwartz, 2022 ) . On the other hand, studying contamination during fine-tuning, as in Yang et al. (2023) , involves substantial training changes—from learning rate to batch size choices— further complicating its generalizability to pre-training. Method Training Process Data Control Model Size Data Size Multilingual Zhou et al. (2023) Extended pre-training ✗ 1.3 B, 7 B NA ✗ Yang et al. (2023) Fine tuning ✗ 7 B, 13 B NA ✗ Magar & Schwartz (2022) Pre-training ✗ 110 M, 345 M 600 M ✗ Jiang et al. (2024) Pre-training ✗ 124 M, 774 M 20 B, 60 B ✗ Ours Pre-training ✓ 1 B, 8 B 325 B ✓ Table 1 : A comparison of related work to our experimental setup. ”Data Control” reflects whether the word checks for existing training-test contamination. Unlike prior work, we study contamination during pre-training in a controlled, large-scale, multilingual setting. Previous works analyze models