Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation
Jonibek Mansurov, Akhmed Sakip, Alham Fikri Aji · 2024
Read Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation by Jonibek Mansurov, Akhmed Sakip, Alham Fikri Aji (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15255.txt.
Introduction
Jonibek Mansurov, Akhmed Sakip, and Alham Fikri Aji expose a trick in a 2024 paper that can fake high scores for language models. The trick passes test lore through a chain of clean-looking training steps. Each step looks like fair work, yet the end model has drunk the test in secret.
With a small two-layer model as the test case, scores on a hard quiz leapt by vast spans with no true gain in thought. The authors warn the trick can be used by plan or by mere chance. A lab may adopt a tainted set in good faith and boast false gains. Graders gain a named flaw to hunt.
Catalog record
| CREATOR | Jonibek Mansurov, Akhmed Sakip, Alham Fikri Aji |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15255
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation Jonibek Mansurov, Akhmed Sakip, Alham Fikri Aji Abstract In this paper, we show that knowledge distillation can be subverted to manipulate language model benchmark scores, revealing a critical vulnerability in current evaluation practices. We introduce "Data Laundering," a process that enables the covert transfer of benchmark-specific knowledge through seemingly legitimate intermediate training steps. Through extensive experiments with a 2-layer BERT student model, we show how this approach can achieve substantial improvements in benchmark accuracy (up to 75% on GPQA) without developing genuine reasoning capabilities. Notably, this method can be exploited intentionally or even unintentionally, as researchers may inadvertently adopt this method and inflate scores without realising the implications. While our findings demonstrate the effectiveness of this technique, we present them as a cautionary tale highlighting the urgent need for more robust evaluation methods in AI. This work aims to contribute to the ongoing discussion about evaluation integrity in AI development and the need for benchmarks that more accurately reflect true model capabilities. The code is available at https://github.com/mbzuai-nlp/data_laundering . \setcode utf8 1 Introduction The increasing reliance on language model benchmarks like MMLU ( Hendrycks et al. 2021 ) , GPQA Rein et al. 2024 , and BigBench ( Srivastava et al. 2023 ) has solidified these metrics as standard measures for assessing and comparing model capabilities, driving innovation and tracking progress in artificial intelligence (AI). However, this focus on benchmark performance has also introduced vulnerabilities, incentivizing potential manipulation and exploitation of these evaluation metrics ( Yang et al. 2023 ; Zheng et al. 2024 ; Balloccu et al. 2024 ) . Our work builds upon growing concerns in the field regarding data contamination and benchmark integrity. Previous studies have shown how proprietary models like GPT-3 and GPT-4 have inadvertently learned from leaked benchmark data, raising alarm about the integrity of closed-source models ( Brown et al. 2020 ; Magar and Schwartz 2022 ; Balloccu et al. 2024 ) . This contamination undermines reliable evaluation, as models trained on leaked data can achieve inflated scores without developing true generalization. Additionally, recent research has demonstrated that detection methods designed to identify data contamination, such as the LM Contamination Index and text overlap metrics ( Sainz et al. 2023 ; Golchin and Surdeanu 2024 ) , may fall short in identifying more subtle forms of benchmark gaming—especially in closed-source models that implement filtering mechanisms to conceal such behavior ( Ippolito et al. 2023 ) . In this paper, we expose a critical vulnerability within current benchmarking practices through a method we term "Data Laundering". Our method "Data Laundering" process uses knowledge distillation ( Hinton et al. 2015 ; Urban et al. 2017 ; Cheng et al. 2020 ) , a technique traditionally intended for model compression and transfer learning, to covertly transfer benchmark-specific knowledge in a staged manner through intermediate training steps. This process, inspired by the phases of financial laundering, involves three steps—placement, layering, and integration—where we intentionally "place" benchmark knowledge into a teacher model trained on test data, "layer" it through legitimate-seeming intermediate training datasets using knowledge distillation, and finally "integrate" the knowledge into the model by evaluating it on the benchmark, thereby making its performance gains appear as genuine skill acquisition. Importantly, researchers can unintentionally use this method, especially if they lack awareness of the training dataset used for the teacher model AI@Meta 2024 ; Achiam et al. 2023 . If researchers use a teacher model for knowledge distillation without realizing it was trained on contaminated data, this can inflate benchmark performance without genuine skill improvements. While prior work has focused on explicit manipulation of evaluation systems, our approach highlights a more disguised form of benchmark gaming that can occur even under seemingly valid training practices. Through this investigation, we aim not to provide a blueprint for manipulation but rather to stimulate a necessary dialogue around evaluation integrity within the AI community. Benchmark systems must evolve to detect more sophisticated forms of gaming and ensure that scores reflect authentic model capabilities rather than superficial improvements. Our contributions are: 1. Demonstrating a novel form of benchmark manipulation that can be employed intentionally or unintentionally through legitimate-appearing training processes; 2. Providing empirical evidence of how knowledge distillation can be used to "launder" benchmark knowledge covertly; 3. Highlighting the limitations of current evaluation frameworks. 2 Related Work 2.1 Data Contamination in Language Models The challenge of data contamination in language models emerged prominently with GPT-3 ( Brown et al. 2020 ) , which pioneered the API-only access model with limited training data disclosure ( Magar and Schwartz 2022 ) . Despite early evidence suggesting significant contamination ( Raffel et al. 2020 ) , GPT-3’s widespread adoption in research often proceeded without adequate consideration of this issue. Recent work has highlighted growing concerns about data contamination in modern language models. As shown by Balloccu et al. 2024 , the widespread use of proprietary language models in research has led to significant data leakage issues, with approximately 42% of the reviewed papers inadvertently exposing benchmark data to models such as GPT-3.5 and GPT-4. This issue has become particularly pressing with the public release of models such as ChatGPT, PaLM 2 ( Anil et al. 2023 ) , and Claude, where the closed-source nature complicates the contamination assessment. Yang et al. 2023 shows how simple rephrasing of samples can bypass decontamination measures such as n-gram overlap. 2.2 Automatic Benchmark and Evaluation Challenges The integrity of language model benchmarks has become a critical concern in the field, especially as the relience on automated evaluation metrics increases. To meet the need for timely assessments of newly released models, platforms such as Chatbot Arena ( Chiang et al. 2024 ) provide human-based evaluation, but gathering statistically significant human feedback can take time. As a result, Dubois et al. 2024 ; Li et al. 2024 ; Zheng et al. 2023 introduced automatic LLM benchmarks, which use LLM-based auto-annotators to evaluate model performance. However, Zheng et al. 2024 demonstrated that even “null models” returning constant outputs could achieve artificially high scores on certain benchmarks by exploiting structural weaknesses in evaluation templates. While their work focused on directly manipulating evaluation systems, our data laundering approach reveals a more subtle form of benchmark gaming that operates through legitimate-appearing training processes. 2.3 Logit-Based Knowledge Distillation Knowledge distillation ( Hinton et al. 2015 ) techniques have traditionally been used for legitimate purposes such as model compression and transfer learning. Recent advancements have introduced various logit distillation approaches tailored for large language models. Reverse KL Gu et al. 2024 has been used to address the "mode-averaging" issue. DistiLLM Ko et al. 2024 suggests blending the logit distributions of the teacher and student models, while SinKD Cui et al. 2024 replaces KL divergence with Sinkhorn Distance. Our work reveals how logit-based knowledge distillation can be repurposed for potentially problematic uses. 3 Methodology Figure 1: The Data Laundering framework parallels traditional money laundering phases: Placement (knowledge acquisition through teacher model), Layering (knowledge transformation through distillation), and Integration (legitimate knowledge verification through benchmark testing). This analogy illustrates how knowledge can be effectively transferred while maintaining clear separation from source domains. Just as money laundering involves transforming "dirty" money into "clean"