When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation
David Tan, Pinzhen Chen, Josef van Genabith, Koel Dutta Chowdhury · 2026
Read When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation by David Tan, Pinzhen Chen, Josef van Genabith, Koel Dutta Chowdhury (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20858.txt.
Introduction
David Tan and three co-authors show how a test set can leak into a model in a subtle way. Their 2026 paper studies contamination in machine translation evaluation. Large language models can be benchmark-contaminated. They score high by recall, not by skill. In multilingual settings, the memorization can even cross into clean languages. The team uses the FLORES-200 translation benchmark as a diagnostic.
They study two multilingual models of similar size. Bloomz was trained on FLORES; Llama serves as an uncontaminated control. Comparing the two exposes cross-direction contamination. Scores inflate in language pairs the test was not meant to touch. The paper warns evaluators that clean test sets may not stay clean. It also gives a method to check for this hidden leak.
Catalog record
| CREATOR | David Tan, Pinzhen Chen, Josef van Genabith, Koel Dutta Chowdhury |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20858
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation David Tan, Pinzhen Chen, Josef van Genabith, Koel Dutta Chowdhury Abstract Large language models (LLMs) can be benchmark-contaminated, resulting in inflated scores that mask memorization as generalization, and in multilingual settings, this memorization can even transfer to “uncontaminated” languages. Using the FLORES-200 translation benchmark as a diagnostic, we study two 7–8B instruction-tuned multilingual LLMs: Bloomz, which was trained on FLORES, and Llama as an uncontaminated control. We confirm Bloomz’s FLORES contamination and demonstrate that machine translation contamination can be cross-directional, artificially boosting performance in unseen translation directions due to target-side memorization. Further analysis shows that recall of memorized references often persists despite various source-side perturbation efforts like paraphrasing and named entity replacement. However, replacing named entities leads to a consistent decrease in BLEU, suggesting an effective probing method for memorization in contaminated models. 1 Introduction † † Code at github.com/Mr-Ao-25/cross-ling-contamination . Large language models (LLMs) typically go through pre-training and fine-tuning stages to become applicable to downstream tasks, where data is a fundamental building block. Despite this, details about training data are often too vague to infer useful information, both for open- and closed-source models. This has negative implications for LLM research, one of which is the reliability of using public benchmarks to evaluate model performance. LLM training data is typically in the magnitude of billions to trillions of tokens ( 7 ; 12 ) . Consequently, public test sets may inadvertently be incorporated into training data ( 29 ) and artificially boost scores—a phenomenon referred to as data contamination ( 18 ) . The problem is even more pressing in multilingual evaluation, where contamination can cross-lingually transfer into languages not seen before ( 32 ) . This work uses machine translation as a diagnostic task to investigate cross-direction data contamination: the artificial boost of unseen language pair scores, resulting from training on language pairs with the same target. We use FLORES200 ( 20 ) , a test suite that supports hundreds of languages with multiway parallelism, yet this wide coverage increases the likelihood of contamination. We explore whether its multiway parallel structure may result in cross-direction transfer of train-test leakage. We benchmark Bloomz-7B1 ( 19 ) , a multilingual LLM with FLORES documented as part of its fine-tuning data, and compare it to Llama-3.1-8B-Instruct ( 8 ) which is reportedly clean. Compared to 17 ’s focus on the contribution of source or target-side text during training, we probe cross-direction contamination in machine translation at inference, specifically whether altered source inputs can still trigger memorized targets. We analyze patterns of contamination through controlled studies and design perturbation-based tests to probe how contamination manifests. We present three findings: 1. We show that contamination happens cross-directionally for machine translation, where contaminated LLMs can perform moderately well in unseen translation directions, mainly due to target-side language memorization. 2. While it is assumed that train-test source input similarity leads to patterns of contamination, we demonstrate that a not-so-similar input may still lead to a model calling memorized text. 3. We discover that replacing named entities consistently decreases BLEU, suggesting an effective method for probing memorization. Figure 1: BLEU for Bloomz (upper left), Llama (lower left) and COMET for Bloomz (upper right), Llama (lower right). All plots have the same sources, targets, and color bar with a blue-red gradient (low-to-high). 2 Experimental Setup Test set and Models We use FLORES200 ( 20 , or FLORES ) as our test set for its wide language coverage and multi-way parallelism. We use the dev set of 997 samples and evaluate the model’s initial generation for each sample. We select 15 languages spanning different resource levels—high-resourced: English ( eng ), Simplified Chinese ( zho ), Spanish ( spa ), Portuguese ( por ), French ( fra ), and Vietnamese ( vie ); medium-resourced: Malayalam ( mal ), Tamil ( tam ), Assamese ( asm ), and Odia ( ory ); low-resourced: Bambara ( bam ), Fon ( fon ), Ewe ( ewe ), Kikuyu ( kik ), and Maori ( mri ). Notably, Bloomz has not been trained on ewe and mri . We investigate various aspects of contamination using bloomz-7b1 ( 19 , Bloomz ) and Llama-3.1-8B-Instruct ( 8 , Llama ) . Both models are instruction-tuned, similarly sized (7–8B parameters), and designed for multilingual text generation, making them broadly comparable in their cross-lingual capabilities. While Bloomz was explicitly trained on FLORES with unspecified translation directions, there is no reported use of FLORES in the training of Llama. For entity replacement experiments, we use aya-expanse-8b ( 3 , Aya ) . 3 Contamination in Machine Translation 3.1 Is there contamination? We first investigate whether Bloomz and Llama have memorized FLORES. We run an evaluation on FLORES for every source-target pair formed by our 15 chosen languages. We use BLEU implemented in sacrebleu ( 24 ; 23 ) as a measure of surface form overlap w.r.t. the reference, rather than translation quality, and COMET-22-DA ( 28 ) for semantic similarity. As our diagnostic, we take that an unreasonably high BLEU coupled with high COMET (e.g. 80 BLEU, 0.9 COMET) would imply contamination, while a reasonable BLEU with a high COMET score (e.g. 40 BLEU, 0.9 COMET) would indicate a good performance without contamination (or with less likelihood of it). Low BLEU and COMET indicate divergence in both form and meaning; moderate BLEU with high COMET reflects semantic similarity despite surface variation. Figure 1 presents BLEU and COMET heatmaps for Bloomz and Llama. For Bloomz, we see oddly high BLEU (40–90) in xxx \to { eng , fra , mal , ory , por , tam , vie , zho }, hinting at contamination; medium BLEU scores ( \leq 50) in xxx \to { asm , spa }; and very low scores ( \approx 0) in xxx \to { bam , ewe , fon , kik , mri }. COMET scores generally follow trends seen in BLEU; language pairs with medium to high BLEU achieve > 0.8 COMET; low BLEU language pairs achieve \leq 0.5 – 0.65 COMET. In contrast, Llama shows little evidence of contamination: reasonably high BLEU scores (30–40) only appear in a few high-resource languages or directions, and COMET scores are only high ( \geq 0.8) when BLEU scores are > 15. For Bloomz, high COMET ( \geq 0.8) results from \geq 35 BLEU. Memorization not generalization To rule out the possibility that Bloomz simply has superb performance in those directions with high BLEU, we test the relatively low-resource eng \to { mal , ory , tam } available in PMIndia 10 and Mann-ki-Baat 15 . Appendix B Table 2 shows near-zero BLEU scores for both datasets, indicating that Bloomz cannot translate those directions, adding further evidence of FLORES contamination. Disentangling source and target We study the pattern of contamination when a language is placed on the source or target side. Figure 1 shows that when placed on the target side, a handful of languages have a “clean” column: bam , ewe , fon , kik , and mri , with BLEU \approx 0. In contrast, when placed on the source side, a few languages that did not exhibit contamination signals as the target ( bam