Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models
Haritz Puerto, Martin Gubri, Sangdoo Yun, Seong Joon Oh · 2024
Read Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models by Haritz Puerto, Martin Gubri, Sangdoo Yun, Seong Joon Oh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00154.txt.
Description
Abstract Membership inference attacks (MIA) attempt to verify the membership of a given data sample in the training set for a model. MIA has become relevant in recent years, following the rapid development of large language models (LLM). Many are concerned about the usage of copyrighted materials for training them and call for methods for detecting such usage.
Catalog record
| CREATOR | Haritz Puerto, Martin Gubri, Sangdoo Yun, Seong Joon Oh |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00154
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models Haritz Puerto, Martin Gubri, Sangdoo Yun, Seong Joon Oh Abstract Membership inference attacks (MIA) attempt to verify the membership of a given data sample in the training set for a model. MIA has become relevant in recent years, following the rapid development of large language models (LLM). Many are concerned about the usage of copyrighted materials for training them and call for methods for detecting such usage. However, recent research has largely concluded that current MIA methods do not work on LLMs. Even when they seem to work, it is usually because of the ill-designed experimental setup where other shortcut features enable ‘‘cheating.’’ In this work, we argue that MIA still works on LLMs, but only when multiple documents are presented for testing. We construct new benchmarks that measure the MIA performances at a continuous scale of data samples, from sentences (n-grams) to a collection of documents (multiple chunks of tokens). To validate the efficacy of current MIA approaches at greater scales, we adapt a recent work on Dataset Inference (DI) for the task of binary membership detection that aggregates paragraph-level MIA features to enable MIA at document and collection of documents level. This baseline achieves the first successful MIA on pre-trained and fine-tuned LLMs. 1 1 1 Our code is available at https://github.com/parameterlab/mia-scaling 1 Introduction Large language models (LLMs) are trained on vast datasets, which providers typically keep private. Data owners are concerned that their copyrighted data might be used in LLM training without explicit consent. Membership Inference Attacks (MIAs) attempt to determine if a specific data sample was used to train a model ( Shokri et al. 2017 ) . These methods are now being applied to LLMs to address the question of potential data misuse ( Shi et al. 2024 ; Meeus et al. 2024a ; Zhang et al. 2024b ; Wang et al. 2024a ; Xie et al. 2024 ) . Figure 1: Focusing on the Right Scale. MIA has traditionally been considered ineffective for LLMs. However, we argue that MIA remains effective for LLMs when applied at a much larger scale, considering significantly longer token sequences. This large-scale MIA is also practically and legally relevant, as copyright is often determined at the document level. The application of MIAs on pretrained LLMs has faced discouraging results so far. Despite the initial optimism from the works of ( Shi et al. 2024 ; Meeus et al. 2024a ; Carlini et al. 2021 ; Mattern et al. 2023 ) , Duan et al. 2024 ; Das et al. 2024 ; Maini et al. 2024 ; Meeus et al. 2024b show later that current methods only achieve near random-guessing performance. Before their work, it was common practice to select true non-members from documents created after the LLM’s cut-off date . They showed that this approach allowed MIA methods to exploit temporal cues, rather than identifying membership through the model’s inherent response characteristics. Duan et al. 2024 ; Das et al. 2024 ; Maini et al. 2024 introduced a new evaluation method based on an independent, identically distributed (IID) split between true members and non-members. Since adopting this method, no further MIA studies on LLMs have shown performance significantly better than random. Reported membership detection has remained below 60% AUROC, close to the random-chance level of 50% AUROC ( Duan et al. 2024 ; Das et al. 2024 ; Maini et al. 2024 ; Xie et al. 2024 ; Zhang et al. 2024b ) . We argue that MIA can still be effective on LLMs, provided it is applied to much longer token sequences than previously considered. Earlier MIA approaches often focused on short sequences of tokens, typically ranging from 128 to 256 tokens ( Xie et al. 2024 ; Duan et al. 2024 ; Shi et al. 2024 ; Wang et al. 2024b ) . This use of n-grams faced criticism because of the significant overlap between members and non-members, making the MIA problem poorly defined ( Duan et al. 2024 ) . Some later studies suggested analysing entire documents instead of n-grams ( Shi et al. 2024 ; Meeus et al. 2024b ) , but even then, performance remained near random ( Meeus et al. 2024b ) . In this work, we demonstrate that MIA approaches begin to show meaningful performance only when applied to much longer token sequences, such as 10K tokens. To show this, we introduce four scales of token sequences: sentences, paragraphs, documents, and datasets, as shown in Figure 1 . We propose MIA evaluation protocols and benchmarks for the binary detection of training data samples given at four different scales. As a baseline, we adapt the Dataset Inference ( Maini et al. 2024 ) method for the MIA task at multiple scales. As a result, our aggregation-based MIA demonstrates significant performance improvements for document sets, achieving AUROC scores of 80% or higher. To explore additional scenarios, we investigate the performance of MIA on fine-tuning data. Our findings show that continual learning MIA is effective collection of documents (AUROC > 88%), while CoT-based fine-tuning works at both the sentence and collection levels. Our contributions are summarized as: 1. We introduce a novel evaluation benchmark and protocol for MIA, covering multiple scales of token sequences. 2. We extend and adapt the first successful MIA Maini et al. 2024 to any data scale, allowing us to conduct a comprehensive analysis of MIA performance in LLMs. 3. We provide additional MIA benchmarks for various LLM fine-tuning scenarios, demonstrating that our method achieves even stronger performance in these contexts. 2 Background & Related Work Membership inference attacks (MIA) aims to prove that a certain data sample belongs to the training set of a model. Yeom et al. 2018 hypothesize that members have a lower loss (perplexity for LLMs) than non-members and based on this propose to use the loss to infer membership. Carlini et al. 2021 build on top of it and propose to use the ratio of the perplexity and the zlib entropy. Shi et al. 2024 propose to compute the average log-likelihood of the tokens with the lowest probabilities based on the assumption that members would have higher probabilities than non-members. Since most LLMs do not have training-test set splits, initial benchmarks and recent works use the knowledge cut-off date to define members and non-members ( Shi et al. 2024 ; Meeus et al. 2024a ) . Works using these evaluation benchmarks show positive results ( Shi et al. 2024 ; Meeus et al. 2024a ; Zhang et al. 2024b ; Wang et al. 2024a ; Xie et al. 2024 ) . However, Duan et al. 2024 ; Das et al. 2024 ; Maini et al. 2024 showed that the evaluation method based on cut-offs is flawed as these cut-offs introduce temporal biases that are easily captured by a bag of words. Therefore, they proposed to use LLMs trained on datasets that contain a random train-test split for members and non-members, like The Pile dataset ( Gao et al. 2020 ) . In this setup, however, MIA only achieves performance barely above random guessing. Maini et al. 2024 show that aggregating MIA scores across