A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models
Bowen Chen, Namgi Han, Yusuke Miyao · 2024
Read A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models by Bowen Chen, Namgi Han, Yusuke Miyao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13475.txt.
Introduction
Bowen Chen, Namgi Han, and Yusuke Miyao re-test whether attacks can tell if text trained a language model in a paper from 2024. Membership inference matters because training data are rarely disclosed. Past studies disagreed wildly, some claiming success and others near-random results. The authors blame single test settings that sample members and non-members unfairly. They rerun thousands of experiments across settings for each attack method.
Most methods did not beat simple baselines statistically. Performance grew with model size and varied by domain. A notable minority of examples were separable, though, and attacks disagreed about which. Choosing the decision threshold proved a hidden difficulty. Text length and difference helped attacks most.
Catalog record
| CREATOR | Bowen Chen, Namgi Han, Yusuke Miyao |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13475
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models Bowen Chen, Namgi Han, Yusuke Miyao Abstract The lack of data transparency in Large Language Models (LLMs) has highlighted the importance of Membership Inference Attack (MIA), which differentiates trained (member) and untrained (non-member) data. Though it shows success in previous studies, recent research reported a near-random performance in different settings, highlighting a significant performance inconsistency. We assume that a single setting doesn’t represent the distribution of the vast corpora, causing members and non-members with different distributions to be sampled and causing inconsistency. In this study, instead of a single setting, we statistically revisit MIA methods from various settings with thousands of experiments for each MIA method, along with study in text feature, embedding, threshold decision, and decoding dynamics of members and non-members. We found that (1) MIA performance improves with model size and varies with domains, while most methods do not statistically outperform baselines, (2) Though MIA performance is generally low, a notable amount of differentiable member and non-member outliers exists and vary across MIA methods, (3) Deciding a threshold to separate members and non-members is an overlooked challenge, (4) Text dissimilarity and long text benefit MIA performance, (5) Differentiable or not is reflected in the LLM embedding, (6) Member and non-members show different decoding dynamics. 1 Introduction Large Language Models (LLMs) Minaee et al. 2024 are trained with terabyte level corpora Chowdhery et al. 2022 that are automatically collected, even the data creators themselves can hardly give instance-level analysis over the collected corpora Biderman et al. 2022 . Such a situation has led to several issues, such as the data leakage of evaluation benchmarks Sainz et al. 2023 and personal information Yao et al. 2024 . Those concerns inspired the research of the Membership Inference Attacks (MIA) in LLMs Hu et al. 2022 . Figure 1: Sample with different settings may result in MIA performance inconsistency. Given a set of examples, MIA focuses on differentiating members (trained) and non-members (untrained) by calculating a feature value for each example and splitting them using a threshold. Generally, those methods focus on observing the outputs of LLMs like generated tokens, probability distributions, losses, etc., and utilize such features to distinguish between members and non-members. Despite their success in previous studies, recent studies have shown that those methods behave nearly randomly in another MIA construction setting, or such benchmarks can be easily cheated Duan et al. 2024 ; Das et al. 2024 . Those negative results raised an inconsistency regarding the performance of MIA methods, e.g., do those MIA methods really work or not ? We see such inconsistency comes from the distribution of the enormous size of the pre-train corpora, which is possible that members and non-members sampled from one setting could be totally different from another setting, leading to inconsistent results. In this study, instead of a single setting, we evaluate MIA methods statistically from multiple perspectives, e.g., the split methods, domains, text length, and model sizes. This led to thousands of MIA experiments for one MIA method and enabled a statistical analysis for MIA methods. Additionally, we conducted an in-depth analysis to study how the text feature, embedding, threshold decision, and decoding dynamics in members and non-members relate to MIA. We found that: (I) MIA performance improves with model size and varies with domains, while most methods do not statistically outperform baselines. (II) While MIA performance is generally low, a notable amount of differentiable member and non-member outliers exist and vary across MIA methods, connecting the inconsistency regarding the MIA performance. (III) The threshold to separate members and non-members changes with model size and domains, raising it as an overlooked challenge when using MIA in real-world. (IV) While the actual relation varies based on MIA methods, MIA performance generally positively relates to text length and text dissimilarity between members and non-members. (V) Whether members and non-members are differentiable is reflected in LLM embedding with an emergent change in a larger model that makes them more separable. Specifically, the last layer embedding used by the current MIA methods actually has a low embedding separability. (VI) Domains with high MIA performance show a faster increase in accumulated entropy difference for members and non-members. 2 Related Works Membership Inference Attacks (MIA) Hu et al. 2022 differentiates the member (trained) and non-member (untrained) data by calculating feature values and deciding a threshold for classification. 2.1 Membership Inference Attack Methods First, we introduce MIA methods used in LLM based on the model’s transparency. Gray-Box Method This method requires the intermediate outputs to be observable, like loss, token probability, etc. Carlini et al. 2021 calculated the loss difference with another reference model with the assumption that if two models are trained under two samples of the same distribution, then the loss of non-members should be significantly different. Mink- k\% Shi et al. 2024 calculates the average log-likelihood of the tokens with the bottom- k\% output probabilities, suggesting non-member text has more outliers and thus higher negative log-likelihood. Zhang et al. 2024b improved Mink- k\% by standardizing with variance and mean. Zhang et al. 2024c compared predicted token probabilities against actual token probabilities from open corpora, in which the member data should have a closer distribution distance. Additionally, some methods alter the input text, like token swapping Ye et al. 2024 or adding prefixes Xie et al. 2024 with the hypothesis that the likelihood of member data should be influenced more by such text alternation. Black-Box Method This method only observes the output tokens from the LLM. Dong et al. 2024 calculated a variant of edit distance with multiple generations from the LLM with the hypothesis that those generations of a member text should have a smaller lexical distance compared to non-member text. Additionally, Kaneko et al. 2024 made a similar hypothesis while they evaluated the semantic similarity using the embedding model. 2.2 Membership Inference Attack Analysis Regarding the MIA analysis, some research Maini et al. 2021 ; Carlini et al. 2022 suggest the MIA difficulty increases with model size and corpora size. Zhang et al. 2024a showed with toy data that it is hard to reliably operate an MIA method under a certain false positive rate. Meeus et al. 2024 found some MIA benchmarks are flawed, which can be easily cheated by just checking the word differences Das et al. 2024 . Duan et al. 2024 evaluated Gray-Box MIA methods in the train and test set of pre-train corpora of an LLM, where they behave almost randomly. Those negative findings show an inconsistency with the performance reported by the previous MIA methods. We assume such inconsistency comes from the sampled member and non-member distribution under different settings, which could be totally different due to the enormous size of corpora, leading to this inconsistency. In this study, instead of using a single setting, we create various settings, leading to thousands of experiments for one MIA method. This allows a statistical-level analysis of MIA methods from multiple perspectives, which shows new findings and connects the MIA performance inconsistency. 3 Experiments Setting Given a model M and set of data X=\{x_{0}\dots x_{n}\} , where each x is a text consisted of \{t_{0}\dots t_{m}\} tokens a