How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence
Hyeong Kyu Choi ∗ {}^{\;*}, Maxim Khanov ∗ {}^{\;*}, Hongxin Wei, Yixuan Li · 2025
Read How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence by Hyeong Kyu Choi ∗ {}^{\;*}, Maxim Khanov ∗ {}^{\;*}, Hongxin Wei, Yixuan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00678.txt.
Description
Hyeong Kyu Choi, Yixuan Li and their co-authors propose the Kernel Divergence Score to measure benchmark contamination in large language models in 2025. When test data leak into training data, scores rise and no longer show true skill on new data.
Their score compares how similar sample embeddings are before and after fine-tuning on the benchmark in question. The logic is that tuning shifts unseen samples more than ones the model has already seen. In controlled tests, the score tracked contamination levels almost perfectly and beat prior methods. Extra checks confirmed which design choices made the score reliable.
Catalog record
| CREATOR | Hyeong Kyu Choi ∗ {}^{\;*}, Maxim Khanov ∗ {}^{\;*}, Hongxin Wei, Yixuan Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00678