Quantifying the Effect of Test Set Contamination on Generative Evaluations
Rylan Schaeffer, Joshua Kazdan, Baber Abbasi, Ken Ziyu Liu, Brando Miranda, Ahmed Ahmed, Abhay Puri, Niloofar Mireshghallah, Sanmi Koyejo · 2026
Read Quantifying the Effect of Test Set Contamination on Generative Evaluations by Rylan Schaeffer, Joshua Kazdan, Baber Abbasi, Ken Ziyu Liu, Brando Miranda, Ahmed Ahmed, Abhay Puri, Niloofar Mireshghallah, Sanmi Koyejo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04301.txt.
Introduction
Rylan Schaeffer and ten co-authors measure what happens when test material leaks into pretraining data and the test itself asks a model to generate answers, in a 2026 arXiv paper. Contamination has been studied mostly on multiple-choice tests. Here the team pretrains language models on web data mixed with copies of the MATH benchmark, varying model size and how many copies leak in.
Performance rises with contamination and size, and their scaling-law analysis yields a surprise: even one leaked copy lets models reach lower loss than clean training should allow. Further training on fresh data washes the effect out, while finetuning can help or hurt depending on the earlier leak. At sampling time, higher temperatures soften the effect, and longer solutions prove far harder to memorize than short ones.
Catalog record
| CREATOR | Rylan Schaeffer, Joshua Kazdan, Baber Abbasi, Ken Ziyu Liu, Brando Miranda, Ahmed Ahmed, Abhay Puri, Niloofar Mireshghallah, Sanmi Koyejo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantifying the Effect of Test Set Contamination on Generative Evaluations Rylan Schaeffer, Joshua Kazdan, Baber Abbasi, Ken Ziyu Liu, Brando Miranda, Ahmed Ahmed, Abhay Puri, Niloofar Mireshghallah, Sanmi Koyejo Abstract As frontier AI systems are pretrained on web-scale data, test set contamination has become a critical concern for accurately assessing their capabilities. While research has thoroughly investigated the impact of test set contamination on discriminative evaluations like multiple-choice question-answering, comparatively little research has studied the impact of test set contamination on generative evaluations. In this work, we quantitatively assess the effect of test set contamination on generative evaluations through the language model lifecycle. We pretrain language models on mixtures of web data and the MATH benchmark, sweeping model sizes and number of test set replicas contaminating the pretraining corpus; performance improves with contamination and model size. Using scaling laws, we make a surprising discovery: including even a single test set replica enables models to achieve lower loss than the irreducible error of training on the uncontaminated corpus. We then study further training: overtraining with fresh data reduces the effects of contamination, whereas supervised finetuning on the training set can either increase or decrease performance on test data, depending on the amount of pretraining contamination. Finally, at inference, we identify factors that modulate memorization: high sampling temperatures mitigate contamination effects, and longer solutions are exponentially more difficult to memorize than shorter ones, presenting a contrast with discriminative evaluations, where solutions are only a few tokens in length. By characterizing how generation and memorization interact, we highlight a new layer of complexity for trustworthy evaluation of AI systems. 1 Introduction Test set contamination – the inclusion of benchmarks in pretraining data – has emerged as a critical threat to the trustworthy evaluation of frontier AI systems ( Sainz et al. 2023 ; Schaeffer 2023 ; Xu et al. 2024a ; Deng et al. 2024a ; Reuel et al. 2025 ) . Evaluation aims to measure generalization on tasks the model has never seen, yet the sheer scale of modern pretraining makes such contamination increasingly likely ( Brown et al. 2020 ; Du et al. 2022 ; Wei et al. 2022a ; Chowdhery et al. 2022 ; Touvron et al. 2023 ) . Prior research has sought to quantify the impact of contamination through two primary lenses. Statistical approaches attempt to detect contamination or estimate its influence by by modifying the test set, for example, by reordering, rephrasing, or replicating benchmark problems, e.g., ( Oren et al. 2023 ; Ni et al. 2025 ; Shi et al. 2024 ; Golchin & Surdeanu 2023 ; Golchin & Surdeanu 2024 ; Roberts et al. 2024 ; Wang et al. 2025 ; Zhang et al. 2024a ) . In comparison, controlled approaches – which offer the most rigorous measurement – intentionally contaminate pretraining corpora to quantify how specific dosages of leakage translate into inflated performance, e.g., ( Magar & Schwartz 2022 ; Jiang et al. 2024 ; Oren et al. 2023 ; Yao et al. 2024 ; Wang et al. 2025 ; Kocyigit et al. 2025 ; Bordt et al. 2025 ) . For a more thorough discussion, please see Appendix A Related Work. Figure 1 : Performance on Generative Benchmarks Increases with Test Set Contamination and Model Size. Schematic: We pretrained compute-optimal language models ( 34 M– 344 M parameters) on corpora containing different replicas of the MATH test set ( 0 – 3162 ). Evaluation used greedy decoding (temperature =0 ). Left: Math Verify scores (top) rise and cross entropy on the test set (bottom) falls as contamination increases, consistent with discriminative evaluations, with a sharp improvement around evaluations, with a sharp improvement around 100 replicas. Right: The ratio in the loss on the MATH test set between R replicas and 0 grows with model size, meaning larger models benefit more from test set contamination for the same number of replicas. While foundational, these investigations have predominantly focused on discriminative benchmarks like classification or multiple-choice question-answering (MCQA). For example, Magar & Schwartz 2022 used SST-2 ( Socher et al. 2013 ) (classification). Jiang et al. 2024 used SST-2 (classification), MMLU ( Hendrycks et al. 2021a ) (MCQA), SQuAD ( Rajpurkar et al. 2016 ) (MCQA), and CNN/Daily Mail (fill-in-the-middle) ( Nallapati et al. 2016 ) . Oren et al. 2023 used 7 MCQA benchmarks and 1 mathematical problem solving benchmark (GSM8K) ( Cobbe et al. 2021 ) , Yao et al. 2024 used 3 MCQA benchmarks while Bordt et al. 2025 used 7 MCQA benchmarks. However, with the rapid advancement of model capabilities and the advent of reasoning models ( OpenAI et al. 2024a ; Google Gemini Team et al. 2025 ; Xu et al. 2025 ) , the field is shifting towards generative benchmarks, or benchmarks for which the model must generate an answer rather than choosing between provided answers. Whether test set contamination has the same effect on generative evaluation as on discriminative evaluations is unclear. Discriminative evaluations typically require the model to place higher probability mass on the correct choice than on a small number of alternative incorrect choices ( Gao et al. 2024 ; Schaeffer et al. 2025d ) , and candidate choices are often only a couple of tokens long. In contrast, generative evaluations require the model to produce solutions spanning tens-to-thousands of tokens without decohering, and introduce new considerations such as the sampling temperature ( Ackley et al. 1985 ) , the sampling algorithm (e.g., top-k ( Fan et al. 2018 ) , top-p ( Holtzman et al. 2020 ) ) and the solution length. In this work, we quantitively study the effects of test set contamination on generative reasoning, focusing on the widely used MATH benchmark ( Hendrycks et al. 2021b ) . We pretrain dozens of language models on corpora contaminated with varying numbers of test set replicas, sweeping across model sizes, sampling temperatures, and token budgets. Our findings reveal that while generative contamination shares some similarities with discriminative settings, it introduces new interactions between memorization and generalization. Figure 2 : Scaling Laws Suggest Including A Single Test Set Replica Achieves Lower Loss Than the Irreducible Error of the Uncontaminated Corpus. Top: For each scaling series pretrained on corpora contaminated with R replicas of the MATH test set, we fit scaling laws \mathcal{L}(C,R)=E(R)+C_{0}(R)\cdot C^{-\alpha(R)} , where C=6\,N\,D is the pretraining compute. Almost all contaminated models achieve lower cross entropy on the MATH test set than the irreducible error of training on uncontaminated data (horizontal purple line). Bottom: Increasing test set contamination reduces the irreducible error E(R) from 3.594 at R=0 to 0.0347 at R=316 . Larger values of R also increase the compute prefactor and compute exponent. The functional form achieves average fitting error <10^{-2} for all R . In this work, we make the following contributions: • Pretraining (Scaling & Irreducible Error): We quantify how contamination impacts the initial pretraining phase. We find that performance scales with contamination frequency and model size, similar to discriminative settings. Using neural scaling laws, we discovered that including even a single replica of the test set enables models to achieve a lower loss than the irreducible error achievable with the uncontaminated corpus. • Further Training (Overtraining & Supervised Finetuning):