Soft Contamination Means Benchmarks Test Shallow Generalization
Ari Spiesberger, Juan J Vazquez, Nicky Pochinkov, Tomáš Gavenčiak, Peli Grietzer, Gavin Leech, Nandi Schoots · 2026
Read Soft Contamination Means Benchmarks Test Shallow Generalization by Ari Spiesberger, Juan J Vazquez, Nicky Pochinkov, Tomáš Gavenčiak, Peli Grietzer, Gavin Leech, Nandi Schoots (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12413.txt.
Description
Benchmark scores are meant to measure how well a model generalises, but training data may already contain test-like material. Ari Spiesberger led this work with six co-authors, studying soft contamination: semantic duplicates of benchmark items that slip past n-gram filters because their wording differs while their content matches. Standard decontamination misses those near-copies.
Embedding the Olmo3 training corpus, the team finds semantic duplicates for 78 percent of CodeForces items and exact duplicates for 50 percent of ZebraLogic problems. Training on such duplicates lifts benchmark scores, and fine-tuning on duplicates also lifts scores on truly held-out items from the same benchmark. Recent benchmark gains, the paper argues, thus mix genuine progress with accumulated test-like data in growing corpora.
Catalog record
| CREATOR | Ari Spiesberger, Juan J Vazquez, Nicky Pochinkov, Tomáš Gavenčiak, Peli Grietzer, Gavin Leech, Nandi Schoots |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-12413