Pitfalls of Evaluating Language Models with Open Benchmarks
Md. Najib Hasan, Mohammad Fakhruddin Babar, Souvika Sarkar, Monowar Hasan, Santu Karmaker · 2025
Read Pitfalls of Evaluating Language Models with Open Benchmarks by Md. Najib Hasan, Mohammad Fakhruddin Babar, Souvika Sarkar, Monowar Hasan, Santu Karmaker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00460.txt.
Introduction
Najib Hasan and co-authors warned in 2025 that open test sets for language models invite cheating. Public benchmarks help the field compare models fairly, but their answers sit in the open for anyone to train on. The team proved the point by fine-tuning small models directly on public test sets. Those models topped the target tests. Yet they failed badly on fresh, similar ones.
The authors then tried simple defenses based on rewording test items, with mixed success. Their lessons are threefold. High scores on a few fixed public tests may not mean real skill. Private or freshly made tests should sit beside open ones. And the field should rethink how it grades models. The paper speaks to anyone who trusts a leaderboard.
Catalog record
| CREATOR | Md. Najib Hasan, Mohammad Fakhruddin Babar, Souvika Sarkar, Monowar Hasan, Santu Karmaker |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00460
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pitfalls of Evaluating Language Models with Open Benchmarks Md. Najib Hasan, Mohammad Fakhruddin Babar, Souvika Sarkar, Monowar Hasan, Santu Karmaker Abstract Open Large Language Model (LLM) benchmarks, such as HELM and BIG-bench, offer standardized, transparent protocols that facilitate the fair comparison, reproducibility, and iterative advancement of Language Models (LMs). However, their openness also introduces critical and underexplored pitfalls. This study exposes these weaknesses by systematically constructing “cheating” models—smaller variants of BART, T5, and GPT-2 fine-tuned directly on public test sets—which achieve top rankings on a prominent open, holistic benchmark (HELM) despite poor generalization and limited practical utility. Our findings underscore three key insights: (a) high leaderboard performance on open benchmarks may not always reflect real-world effectiveness; (b) private or dynamic benchmarks must complement open evaluations to safeguard integrity; and (c) a fundamental reevaluation of current benchmarking practices is essential to ensure robust and trustworthy LM assessments. 1 Introduction The rapid advancement of Large Language Models (LLMs) has raised critical questions regarding their fair and reliable evaluation. To address this important issue, several holistic benchmarks have been developed, for example, Google introduced BIG-bench Srivastava et al. 2022 , Stanford introduced HELM Liang et al. 2022 , Anthropic developed the HH-RLHF dataset Bai et al. 2022 , and OpenAI proposed TruthfulQA Lin et al. 2022 . These benchmarks offer standardized, transparent protocols that facilitate fair comparisons, reproducibility, and iterative advancement of Language Models (LMs). However, despite their many strengths, these open benchmarks also have critical pitfalls that have been largely overlooked in the current literature. The openness that enables transparency and reproducibility also renders these benchmarks susceptible to exploitation. In this study, we expose the pitfalls of the open-evaluation benchmark by constructing “cheating” models—smaller variants of BART Lewis et al. 2020 , T5 Raffel et al. 2020 , and GPT-2 Radford et al. 2019 —fine-tuned directly on a public testing benchmark called HELM (Holistic Evaluation of Language Models) Liang et al. 2022 . This paper focuses exclusively on HELM, without loss of generality, as the target benchmark due to its wide popularity in LLM research. 1 1 1 However, our claims are generalizable and should hold across other relevant benchmarks. HELM includes 42 diverse scenarios across domains such as medicine, law, commonsense reasoning, among others. It promotes transparency through a public leaderboard and by releasing its complete evaluation pipeline, including datasets, metrics, and code. To enhance usability, Liang et al. 2022 also introduced HELM-lite, a compact suite of 10 core scenarios (e.g., MMLU Hendrycks et al. 2020 , MedQA Jin et al. 2020 , NarrativeQA, OpenBookQA Mihaylov et al. 2018 ) that enables efficient experimentation while preserving alignment with the full framework. Our experiments used HELM-lite to study two core research questions. • RQ 1. To what extent can small cheating language models game the HELM leaderboard? • RQ 2. Do high scores in HELM scenarios reflect the true capabilities of language models? To answer these questions, we conduct extensive experiments that reveal the following key insights. • High scores on open benchmarks often reflect test set memorization (also known as “leakage”) rather than true generalization. • Full transparency, while valuable, introduces the risk of “cheating” and compromises evaluation integrity. Hence, open benchmarks must be paired with private or dynamic evaluations to ensure reliability and robustness. • A fundamental rethinking of how the community approaches evaluation and benchmarking is warranted, as the openness designed to promote fairness can inadvertently undermine the reliability of leaderboards. 2 Related Work Early NLP benchmarks focused on isolated tasks, limiting their ability to assess a broad range of LM capabilities. Models like GPT-3 Brown et al. 2020 and T5 Raffel et al. 2020 exposing the limitations of older benchmarks Raji et al. 2021 ; McCoy et al. 2019 . Recently, several holistic benchmarks have been proposed to assess the diverse capabilities of LLMs. BIG-bench Srivastava et al. 2022 was designed to test LLMs on 200+ tasks. However, using public web-based content leads to data leakage, which undermines evaluation integrity and makes it difficult to determine whether high scores come from true generalization or memorization Liang et al. 2022 . TruthfulQA Lin et al. 2022 is a benchmark designed to evaluate the truthfulness and factual accuracy of language model outputs under adversarial-style prompts; however, the benchmark design may not reliably correlate truthfulness with model capabilities Zhou and Shokri 2023 . HELM Liang et al. 2022 addresses these issues. However, HELM makes the entire evaluation pipeline, including test data, public. While this openness promotes transparency, as we shall see in this paper, this allows “cheating” models to overfit directly on evaluation sets, potentially misleading leaderboard rankings. Unlike prior studies that focus on LLM capabilities, our research analyzes open benchmarks from an adversarial perspective. Our study is the first to systematically assess gaming prospects using “cheating” models across open evaluations , highlighting the tension between transparency and evaluation integrity. 3 Experimental Setup 3.1 Scenario and Evaluation Metric We use HELM-lite for our experiments because of its comprehensive evaluation criteria and commitment to transparency. We select HELM-lite as it is more computationally manageable than the entire 42-scenario HELM suite. Although we tested cheating models on HELM-lite, our methodology is broadly generalizable and can be extended to other open benchmarks with a similar setup. HELM-lite contains ten representative scenarios that cover multiple domains as follows: (a) general knowledge (MMLU Hendrycks et al. 2020 , OpenbookQA Mihaylov et al. 2018 ), (b) science (NQ-Open and NQ-Closed Kwiatkowski et al. 2019 ), (c) medicine (MedQA Jin et al. 2020 ), (d) law (LegalBench Guha et al. 2023 ), (e) narrative comprehension (NarrativeQA Kočiský et al. 2018 ), (f) translation (WMT-2014 Bojar et al. 2014 ), and (g) mathematical reasoning (GSM8K Cobbe et al. 2021 , MATH Hendrycks et al. 2021 ). Each scenario is evaluated using metrics defined in the original HELM benchmark. For training our “cheating” models, we use the test set from each of these ten scenarios. To maintain consistency, we follow the same evaluation setup as HELM-lite. 3.2 Models In this study, we focus on lightweight language models—each with fewer than 250 million parameters—to investigate whether smaller architectures can effectively “game” the HELM-lite benchmark. Our selection includes two encoder–decoder models (BART Lewis et al. 2020 and T5 Raffel et al. 2020 ), and one decoder-only model (GPT-2 Radford et al. 2019 ). Across these three families, we evaluate six model configurations to assess how architectural depth and parameter size impact overfitting behavior. For BART, we begin with a mid-sized variant consisting of 3 encoder and 3 decoder layers. Based on its performance, we expand the analysis upward to the full BART-base (6 encoder and 6 decoder) and downward to a minimal BART (1 encoder and 1 decoder) model. We follow a similar approach with the T5 family, evaluating both T5-base (12 encoder - 12 decoder) and its smaller variant, T5-small (6 encoder - 6 decoder), both of which showed strong results. For the GPT family, we use GPT-2 (12 decoder layers), restricting our evaluation to this one configuration due to computational limits. Table 1 summarizes these configurations. Each model is trained using specific hyperparameter settings. Table 3 (see Appendix) lists the hyperparameters. Model Family # of Layers (Encoder / Decoder) #