How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence

Hyeong Kyu Choi ∗ {}^{\;*}, Maxim Khanov ∗ {}^{\;*}, Hongxin Wei, Yixuan Li · 2025

Read How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence by Hyeong Kyu Choi ∗ {}^{\;*}, Maxim Khanov ∗ {}^{\;*}, Hongxin Wei, Yixuan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00678.txt.

Description

Hyeong Kyu Choi, Yixuan Li and their co-authors propose the Kernel Divergence Score to measure benchmark contamination in large language models in 2025. When test data leak into training data, scores rise and no longer show true skill on new data.

Their score compares how similar sample embeddings are before and after fine-tuning on the benchmark in question. The logic is that tuning shifts unseen samples more than ones the model has already seen. In controlled tests, the score tracked contamination levels almost perfectly and beat prior methods. Extra checks confirmed which design choices made the score reliable.

Catalog record

CREATORHyeong Kyu Choi ∗ {}^{\;*}, Maxim Khanov ∗ {}^{\;*}, Hongxin Wei, Yixuan Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-00678

Browse the full library · About the Tower of Babel