C 2 LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation

Yanyang Li, Tin Long Wong, Cheung To Hung, Jianqiao Zhao · 2024

Read C 2 LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation by Yanyang Li, Tin Long Wong, Cheung To Hung, Jianqiao Zhao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04947.txt.

Introduction

The authors present C2LEVA in a 2024 paper on comprehensive, contamination free evaluation of language models. Benchmark scores can be inflated when test data leak into training data, and builders rarely reveal their training sets. Trustworthy testing needs tasks that renew themselves and stay protected.

Their bilingual benchmark covers twenty two tasks, each aimed at a specific ability or application. A systematic strategy automates test data renewal and enforces protection during release. Fifteen open and proprietary models were evaluated at scale. The authors argue the design gives a fuller and more honest picture of model ability.

Catalog record

CREATORYanyang Li, Tin Long Wong, Cheung To Hung, Jianqiao Zhao
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04947

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

C 2 LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation Thanks: Corresponding author. Yanyang Li, Tin Long Wong, Cheung To Hung, Jianqiao Zhao Abstract Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns, particularly regarding data contamination due to the lack of access to proprietary training data. To address this issue, we present C 2 LEVA , a comprehensive bilingual benchmark featuring systematic contamination prevention. C 2 LEVA firstly offers a holistic evaluation encompassing 22 tasks, each targeting a specific application or ability of LLMs, and secondly a trustworthy assessment due to our contamination-free tasks, ensured by a systematic contamination prevention strategy that fully automates test data renewal and enforces data protection during benchmark data release. Our large-scale evaluation of 15 open-source and proprietary models demonstrates the effectiveness of C 2 LEVA 1 1 1 https://github.com/LaVi-Lab/C2LEVA . 1 Introduction Data contamination Brown et al. (2020) ; Liang et al. (2022) , where test data appears in the training set, has become the central concern in delivering trustworthy evaluations for large language models (LLMs), as these models are typically trained on large-scale corpora that are poorly understood Dodge et al. (2021) . A line of research in data contamination focuses on preventing test data leakage before the evaluation. Common practices include concealing the entire Sun et al. (2023) or part Li et al. (2023) of the test set during benchmark releases. However, these approaches tend to lose effectiveness over time Jacovi et al. (2023) . Recently, more promising prevention methods that rely on renewing test data have emerged. These strategies do not depreciate over time. For instance, LatestEval Li et al. (2024) and LiveBench White et al. (2024) gather up-to-date text from the web to create new test cases. DyVal Zhu et al. (2023a) , S3Eval Lei et al. (2023) and NPHardEval Fan et al. (2023) synthesize new test data. Additionally, Wang et al. (2024) ; Ying et al. (2024) ; Qian et al. (2024) generate new test cases from existing data. Despite these advances, two challenges persist: Missing a comprehensive task taxonomy. Most prevention methods target only a limited number of tasks. For example, LatestEval Li et al. (2024) focuses solely on reading comprehension problems. DyVal Zhu et al. (2023a) , S3Eval Lei et al. (2023) and NPHardEval Fan et al. (2023) only evaluate LLMs exclusively on reasoning tasks. LiveBench White et al. (2024) is relatively comprehensive, covering 18 tasks across 6 categories, but still omits crucial tasks such as those related to harms and practical use cases Liang et al. (2022) . There is a clear need for a comprehensive, contamination-free benchmark to evaluate LLMs holistically. Overlooking the contamination risk. Even though existing prevention methods claim to be free of contamination, they often overlook that “new” does not always imply “unseen”. Users can repurpose open data Jacovi et al. (2023) , making contamination possible even in continuously updated benchmarks. Moreover, these prevention approaches are typically “passive”, lacking control over the released data. Some users may intentionally train their models on the test set and cheat on the benchmark inconspicuously until new data is released. For tasks involving expensive or scarce data sources, such as those related to human values or knowledge, existing methods may soon lose their effectiveness. We present C 2 LEVA , a benchmark toward C omprehensive and C ontamination-free L anguage model EVA luation that addresses the aforementioned issues with the following features: • Systematic Contamination Prevention. C 2 LEVA systematically prevents data contamination from both the passive and active perspectives: the passive solution aligns with existing work by updating benchmark data to ensure uncontaminated evaluation. We specifically address repurposing attacks through contamination detection and mitigate data scarcity via data augmentation. The novel active solution minimizes unauthorized use of test data by implementing data protection techniques Wei et al. (2024) ; Zhao et al. (2024) during benchmark release, thereby prolonging the effectiveness of the passive solution. To the best of our knowledge, we are the first to propose active prevention for data contamination, instantiated with data protection. • A Comprehensive and Contamination-Free Task Taxonomy. To ensure extensive coverage, we follow the task taxonomy of Li et al. (2023) and apply our contamination prevention techniques to its critical tasks. C 2 LEVA contains 22 tasks for application assessment and ability evaluation : application assessment encompasses core scenarios of Liang et al. (2022) , while ability evaluation gauges LLM capabilities across four aspects: language, knowledge, reasoning, and harms. Additionally, C 2 LEVA provides at least 5 prompt templates for each task to mitigate prompt sensitivities Zhu et al. (2023b) , contributing to a robust evaluation. Furthermore, C 2 LEVA offers data in both English and Simplified Chinese, facilitating the understanding of cross-lingual transfer in LLMs. C 2 LEVA is thoroughly validated through a large-scale evaluation of 15 open-source and proprietary LLMs. The corresponding leaderboard will be continuously maintained and updated with new evaluation results for emerging models and data. Our experiments also reveal the limitations of the current data protection method in preventing data contamination, underscoring the need for improved approaches in this new research area. 2 Related Work Existing work in data contamination can be divided into two main categories Jacovi et al. (2023) : reactive approaches aim to detect potential contamination risks in the evaluation results of existing benchmarks and models, while preventative approaches target preventing contamination before the evaluation. Reactive Contamination Detection. Contamination detection is an application of membership inference attacks Shokri et al. (2017) ; Yeom et al. (2018) , which aim to determine whether an arbitrary sample is part of a given model’s training data. There are numerous works on contamination detection, typically based on various assumptions: if the training data is available, N-gram matching Brown et al. (2020) ; Dodge et al. (2021) is the most popular approach to report the contamination risk, despite its vulnerability to rephrasing Yang et al. (2023b) . If only a white-box model is available, most methods exploit token probabilities for accurate contamination detection Shi et al. (2023b) ; Oren et al. (2023) ; Zhang et al. (2024) . If we can only access a text-generation API, detection techniques like prompting Golchin and Surdeanu (2023) and synthetic data Wei et al. (2023) ; Duarte et al. (2024) are proposed. In this work, we do not focus on proposing new detection methods but rather treat existing detection methods as a building block in preventative contamination mitigation. Preventative Contamination Mitigation. Hiding the test set completely Sun et al. (2023) or partially Li et al. (2023) has been a common practice to prevent data contamination. However, this approach faces challenges due to the repurposing of test data and difficulties in maintenance Jacovi et al. (2023) . Recent methods seek to maintain the trustworthiness of evaluation results by continuously updating the data. Some work constructs new test cases from the latest web data Li et al. (2024) ; White et al. (2024) . Specifically, for reasoning tasks that can be characterized by rules, various systems have been proposed to synthesize data for evaluation Zhu et al. (2023a) ; Lei et al. (2023) ; Fan et al. (2023) . However, these methods are vulnerable if

Read the complete text