CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
Takashi Ishida, Thanawat Lodkaew, Ikko Yamane · 2025
Read CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting by Takashi Ishida, Thanawat Lodkaew, Ikko Yamane (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18102.txt.
Introduction
Takashi Ishida, Thanawat Lodkaew, and Ikko Yamane build an alarm for test set gaming into language model benchmarks in a 2025 paper. Posting a benchmark with its answers invites trouble: builders may train on it or tune to it, and scores stop meaning skill. Keeping tests private only moves the gaming into feedback loops with the hosts.
Their method, CapBencher, hides the true answer key by design. For each item it writes several answers that are all sound, then posts just one as the key. That caps the best score any honest model can reach. A model that beats the cap has almost surely seen the key. Tests across benchmarks and training styles show the alarm trips on overfit runs and stays quiet on clean ones.
Catalog record
| CREATOR | Takashi Ishida, Thanawat Lodkaew, Ikko Yamane |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18102
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting Takashi Ishida, Thanawat Lodkaew, Ikko Yamane Abstract Publishing a large language model (LLM) benchmark (especially its ground-truth answers) on the Internet risks contaminating future LLMs and enabling evaluation gaming: it may be unintentionally (or intentionally) used to train or select a model, or exploited to overfit and hack leaderboards when labels are accessible. A common mitigation is to keep the benchmark private and let participants submit their models or predictions to the organizers, but this still permits test-set overfitting through feedback loops. To overcome this issue, we propose CapBencher, a way to publish benchmarks without fully disclosing the ground-truth answers, while preserving open evaluation of LLMs. The main idea is to reduce the best possible accuracy, i.e., Bayes accuracy, by injecting randomness to the answers by preparing several logically correct answers, and only include one of them as the solution in the benchmark. Not only does this obscure the ground-truth answers, but it also offers a test for leakage or gaming: since even fully capable models should not surpass the Bayes accuracy, any model that does is a strong signal. We show theoretically and empirically that CapBencher accurately detects test-set overfitting across diverse benchmarks, models, training methodologies, and scenarios. 1 Introduction Figure 1 : CapBencher overview. Before publishing benchmarks, our method reduces the Bayes accuracy, i.e., “caps” the accuracy, which obscures the ground-truth answers. After this modified benchmark is published on the Internet, it may potentially contaminate the training set of LLMs. If contaminated, we expect the LLM to have a higher accuracy than the Bayes accuracy. Constructing modern large language model (LLM) benchmarks has become increasingly expensive as we aim to assess ever-higher capabilities of frontier LLMs. For instance, FrontierMath ( Glazer et al., 2024 ) is a dataset of exceptionally challenging math problems created with more than 60 professional mathematicians, including IMO gold medalists and a Fields Medalist. The Humanity’s Last Exam ( Phan et al., 2025 ) is an academic benchmark assembled by over 1,000 expert contributors across 50 countries. These examples show that modern benchmark creation has become a resource-intensive endeavor requiring far more effort than earlier datasets. Publishing ground-truth answers on the Internet risks contaminating future models or incentivize test-set overfitting, and undermines the value of this intensive effort. A common mitigation is to (partially) keep it private, with participants submitting their models or prediction files to the organizers, who run them on the hidden test set and return the official score ( Wang et al., 2019 ; Zhang et al., 2024a ; Phan et al., 2025 ) . However, this strategy does not offer a solid detection method against test-set overfitting through repeated queries, i.e., leaderboard hacking. To overcome this issue, we propose a protocol to publish benchmarks while obfuscating (rather than fully revealing) the ground-truth answers to the questions, maintaining the ability to openly evaluate LLMs and enabling detection of contamination or gaming. Our idea is to inject uncertainty to the answers by preparing several logically correct answers and selecting one of them at random as the solution in the benchmark. Fig. 1 shows an example where the original question and answer (top left) are transformed into randomized ones (top right). The uncertainty introduced by this modification superficially changes the best-possible accuracy, i.e., the Bayes accuracy, but not the core challenges in answering the questions. Indeed, we demonstrate theoretically and empirically that we can still evaluate and rank LLMs with the modified version of the benchmark similarly to the way the original version would. Our methodology is flexible and can be applied to many different types of benchmarks including direct answer tasks, multiple-choice tasks, and some types of open-ended tasks: it is applicable as long as we can provide multiple correct answers with randomness. A notable advantage of this approach is that we can naturally detect data contamination. If our training dataset was contaminated with many of these randomized questions and answers, the LLM will try to memorize the realized values, e.g., 19 in Fig. 1 , and the accuracy of the LLM will likely surpass 50%. We call this procedure as CapBencher , because we are capping the model accuracy with a ceiling, e.g., 50%. To summarize, our main contribution is proposing a novel way to publish LLM benchmarks and a framework to detect dataset contamination or leaderboard hacking, by explicitly reducing the Bayes accuracy. We theoretically and empirically show that our method can still evaluate and track the improvements of LLMs, while obfuscating the ground-truth answers. We show empirical evidence that our method can detect leakage accurately on a wide range of benchmarks, models, and training methodologies such as pretraining, continual pretraining, and model merging. Furthermore, we demonstrate how our method can be used for private held-out sets (if we do not want to publish even the questions), where we can detect test-overfitting when repeated queries are sent for evaluation. Remark about related work A large body of work studies contamination detection for LLMs, but they typically require access to model probabilities or internals ( Golchin and Surdeanu, 2024 ; Oren et al., 2024 ; Shi et al., 2024 ; Yax et al., 2025 ) or to the exact benchmark text at evaluation time ( Golchin and Surdeanu, 2023 ) , and some can easily be circumvented, e.g., by removing canaries. In contrast, CapBencher gives benchmark creators a way to publish questions while obfuscating the ground - truth answers ; evaluation remains fully open, and works with black - box models. Empirically, we also find that CapBencher detects contamination more reliably than state-of-the-art (SoTA) baselines, e.g., Min - k%/Min - k% ++ ( Shi et al., 2024 ; Zhang et al., 2025 ) . At a glance our idea might resemble adding label noise. However, label noise corrupts correctness and reduces the benchmark’s ability to separate models, often obscuring progress ( Vendrow et al., 2025 ) ; CapBencher preserves logical correctness and model ranking remains consistent. While dynamic benchmarks and evaluation ( Sculley et al., 2025 ; Chen et al., 2025 ) are emerging as a SoTA approach, they are still limited to certain domains or relatively simple problems that can be generated and solved by known, efficient algorithms; CapBencher can be applied to any problems with no known efficient algorithms as long as answers are provided, e.g., by human experts. CapBencher is also inspired by the classical notion of Bayes error ( Cover and Hart, 1967 ) and a line of work on estimating it ( Fukunaga and Hostetler, 1975 ; Theisen et al., 2021 ; Ishida et al., 2023 ; Jeong et al., 2024 ; Ushio et al., 2026 ) ; a key distinction is that, as benchmark designers, we can directly control the Bayes accuracy. See App. A for a detailed comparison to various related topics. 2 Method We consider a question space \mathcal{Q} and an answer space \mathcal{A} ; since an LLM answers in text, these can be all finite-length token sequences. Question–answer pairs are drawn from a joint distribution P_{X,Y} over \mathcal{Q}\times\mathcal{A} , where X is a question and Y its answer. Given a test dataset \{(x_{i},y_{i})\}_{i=1}^{n} sampled from P_{X,Y} , we test an LLM