Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning

Rongzhe Wei, Mufei Li, Mohsen Ghassemi, Eleonora Kreačić, Yifan Li, Xiang Yue, Bo Li, Vamsi K. Potluru, Pan Li, Eli Chien · 2024

Read Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning by Rongzhe Wei, Mufei Li, Mohsen Ghassemi, Eleonora Kreačić, Yifan Li, Xiang Yue, Bo Li, Vamsi K. Potluru, Pan Li, Eli Chien (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08559.txt.

Introduction

Rongzhe Wei and co-authors warn that unlearning tests hide risks for minority data in a 2024 paper. Language models memorize private human text, so methods try to erase chosen data after training. Certified erasure is too strict for these models, leaving heuristic methods judged by attack tests.

Standard tests delete random data, unlearn, then probe with membership attacks against a retrained model. Random picks, though, miss the subsets that face the highest risk. Outlier data, often from minority groups, can stay exposed after a passing grade. The authors argue each data point deserves worst-case scrutiny. Fair privacy means the rarest voice in the data is protected too.

Catalog record

CREATORRongzhe Wei, Mufei Li, Mohsen Ghassemi, Eleonora Kreačić, Yifan Li, Xiang Yue, Bo Li, Vamsi K. Potluru, Pan Li, Eli Chien
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08559

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning Rongzhe Wei, Mufei Li, Mohsen Ghassemi, Eleonora Kreačić, Yifan Li, Xiang Yue, Bo Li, Vamsi K. Potluru, Pan Li, Eli Chien Abstract Large Language Models (LLMs) embed sensitive, human-generated data, prompting the need for unlearning methods. Although certified unlearning offers strong privacy guarantees, its restrictive assumptions make it unsuitable for LLMs, giving rise to various heuristic approaches typically assessed through empirical evaluations. These standard evaluations randomly select data for removal, apply unlearning techniques, and use membership inference attacks (MIAs) to compare unlearned models against models retrained without the removed data. However, to ensure robust privacy protections for every data point, it is essential to account for scenarios in which certain data subsets face elevated risks. Prior research suggests that outliers, particularly including data tied to minority groups, often exhibit higher memorization propensity which indicates they may be more difficult to unlearn. Building on these insights, we introduce a complementary, minority-aware evaluation framework to highlight blind spots in existing frameworks. We substantiate our findings with carefully designed experiments, using canaries with personally identifiable information (PII) to represent these minority subsets and demonstrate that they suffer at least 20% higher privacy leakage across various unlearning methods, MIAs, datasets, and LLM scales. Our proposed minority-aware evaluation framework 1 1 1 Code source: https://github.com/Graph-COM/Minority_Aware_LLM_Unlearning.git marks an essential step toward more equitable and comprehensive assessments of LLM unlearning efficacy. 1 Introduction Large Language Models (LLMs) are trained on vast and diverse datasets, often sourced from public content on the web, much of which is generated by humans ( Touvron et al., 2023 ; Ouyang et al., 2022 ) . This practice raises significant ethical concerns, particularly when the data includes sensitive information, leading to potential privacy violations. Individuals whose data has been used may seek to exercise their “right to be forgotten”, a protection guaranteed by regulations such as the General Data Protection Regulation (GDPR) ( Krzysztofek, 2018 ) . The ideal approach to fulfilling such a request is to retrain the LLM from scratch, excluding the data to be removed. However, this solution is prohibitively expensive and impractical for large-scale models. To address this, the concept of machine unlearning has emerged as a promising alternative. Machine unlearning seeks to efficiently modify the LLM so that it becomes statistically indistinguishable from a model retrained from scratch. In this way, no adversary could confidently determine whether a model has undergone an unlearning process or been retrained, ensuring compliance with the “right to be forgotten”. Unfortunately, it remains an open problem to enforce the formal unlearning guarantee for deep neural networks and LLMs without exact retraining. Despite recent progress in theoretical unlearning research ( Guo et al., 2020 ; Sekhari et al., 2021 ; Neel et al., 2021 ; Ullah et al., 2021 ; Chien et al., 2023 ; Ullah & Arora, 2023 ; Chien et al., 2024a ; Chien et al., 2024b ) , their restrictive assumptions limit practical applicability to deep neural networks and LLMs. Concurrently, researchers have developed efficient unlearning heuristics and empirically evaluated their efficacy ( Golatkar et al., 2020a ; Golatkar et al., 2020b ; Graves et al., 2021 ; Liu et al., 2024a ; Liu et al., 2024c ; Yao et al., 2024 ) , often by comparing approximately unlearned models to those retrained from scratch ( Pawelczyk et al., 2024a ) . Among the various evaluation methods, membership inference attacks (MIAs) ( Shokri et al., 2017 ) , originally developed to infer data usage during training, have been widely adopted for assessing unlearning performance ( Shi et al., 2024b ) . We identify a critical pitfall in the aforementioned LLM unlearning efficacy evaluation. Prior work has demonstrated that unlearning difficulty can vary substantially across individual data points in vision tasks ( Thudi et al., 2024 ; Zhao et al., 2024 ) . Similarly, recent studies on LLMs also reveal highly non-uniform memorization patterns, where certain samples are memorized much more strongly than others ( Feldman & Zhang, 2020 ; Carlini et al., 2022 ) . However, current unlearning evaluation methods only capture “average-case” performance through random data removal from the training set. This approach inadequately addresses privacy risks for hard-to-unlearn data, failing to account for challenging scenarios necessitating rigorous privacy protection ( Steinke & Ullman, 2020 ; Aerni et al., 2024 ) . It neglects the principle that every individual’s right to be forgotten should be upheld equally, thus ignoring data from minority groups, which are often treated as outliers and can be more resistant to unlearning due to the aforementioned stronger memorization effects ( Carlini et al., 2022 ; Nasr et al., 2021 ; Nasr et al., 2023 ) . Consequently, standard unlearning evaluation significantly underestimates privacy risks for these groups, overlooking crucial social responsibilities in personal data protection. Contributions. Motivated by the privacy auditing literature ( Jagielski et al., 2020 ; Steinke et al., 2024 ) , we conduct a synthetic experiment on unlearning injected canaries pertaining to minority groups. We choose Personally Identifiable Information (PII) as a representative minority identifier, while noting that our approach extends to broader cases. We show that minorities suffer from at least 20\% more privacy leakage in most cases across combinations of six unlearning approaches, three MIA variants, three datasets, and two LLMs of different scales. These results underscore the prevalence of the issue in practical settings, highlighting the need for a more effective LLM unlearning evaluation, particularly in regard to privacy risks for minority groups. Accordingly, we propose a minority-aware LLM unlearning evaluation protocol (Figure 1 ) as an initial step toward this goal. With this minority-aware protocol, we benchmark existing unlearning approaches and investigate the effects of forget set size as well as unlearning complexity. This study provides a more holistic understanding of different LLM unlearning approaches for practitioners. Notably, we observe that Langevin Unlearning—the only approach incorporating noise—achieves a favorable privacy-utility trade-off compared to noiseless methods such as SCRUB and Gradient Ascent (GA), suggesting a potential crucial role of noise incorporation in effective unlearning. In summary, these insights underline the critical role of our minority-aware evaluation framework in advancing equitable assessments of unlearning efficacy across different methods. Figure 1 : Illustration of our proposed LLM unlearning approaches (highlighted in red), when compared with the existing pipeline. Standard LLM unlearning evaluation typically involves randomly sampling data for removal from the training set (Case 1), which may underestimate privacy leakage for minority groups. In contrast, we design experiments to assess unlearning efficacy by removing canaries (deliberately inserted data points) related to minority groups (Case 2) and by directly removing data from minority groups (Case 3). Our approach provides a more comprehensive, minority-aware evaluation by considering the worst result across the three settings. 2 Related Work Privacy auditing is a fundamental yet challenging aspect of LLM unlearning due to the difficulty of distinguishing training samples effectively ( Duan et al., 2024 ) . Various privacy-related metrics, such as exposure ( Carlini et al., 2019 ) , mean reciprocal rank ( Wu et al., 2023 ) , extraction likelihood ( Jang et al., 2022 ) , and truth ratio ( Maini et al., 2024 ) , have been proposed to probe privacy

Read the complete text