Auditing Approximate Machine Unlearning for Differentially Private Models
Yuechun Gu Jiajie He Keke Chen · 2025
Read Auditing Approximate Machine Unlearning for Differentially Private Models by Yuechun Gu Jiajie He Keke Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18671.txt.
Introduction
Yuechun Gu and co-authors published a research paper in 2025 on auditing machine unlearning, the process that erases specific data's influence from a trained model. Most checks look only at the erased records, assuming the rest are untouched. A known onion effect suggests peeling one layer can expose others, above all in privacy-guarded models.
The team sets privacy criteria for both erased and retained records, judged by formal privacy math and by membership guessing attacks. They built a cheaper guessing attack that uses data augmentation to cut the cost of shadow models. Their audits found today's unlearning can quietly weaken retained records' privacy. They call for unlearning methods that are private by design.
Catalog record
| CREATOR | Yuechun Gu Jiajie He Keke Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-18671
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Auditing Approximate Machine Unlearning for Differentially Private Models Yuechun Gu Jiajie He Keke Chen Abstract Approximate machine unlearning aims to remove the effect of specific data from trained models to ensure individuals’ privacy. Existing methods focus on the removed records and assume the retained ones are unaffected. However, recent studies on the privacy onion effect indicate this assumption might be incorrect. Especially when the model is differentially private, no study has explored whether the retained ones still meet the differential privacy (DP) criterion under existing machine unlearning methods. This paper takes a holistic approach to auditing both unlearned and retained samples’ privacy risks after applying approximate unlearning algorithms. We propose the privacy criteria for unlearned and retained samples, respectively, based on the perspectives of DP and membership inference attacks (MIAs). To make the auditing process more practical, we also develop an efficient MIA, A-LiRA, utilizing data augmentation to reduce the cost of shadow model training. Our experimental findings indicate that existing approximate machine unlearning algorithms may inadvertently compromise the privacy of retained samples for differentially private models, and we need differentially private unlearning algorithms. For reproducibility, we have pubished our code: 1 1 1 https://anonymous.4open.science/r/Auditing-machine-unlearning-CB10/README.md I Introduction The “right to be forgotten” allows data contributors to request the deletion of their data from an organization’s records. Recent regulations, including the General Data Protection Regulation (GDPR) in the European Union [ 12 ] , the California Consumer Privacy Act (CCPA) in the U.S. [ 5 ] , and Canada’s Personal Information Protection and Electronic Documents Act (PIPEDA) [ 27 ] , have solidified this right. Consumers are also actively exercising this privacy right. For example, Google received over 3.2 million requests to remove specific URLs from search results over five years, demonstrating the importance and scale of this issue. For machine learning models, exercising the right to be forgotten entails not only removing the affected training data samples but also ensuring models are retrained with the updated dataset. However, with the rise of large models, frequent model retraining brings unbearable costs to model owners. Thus, the approaches to avoid high complete retraining costs, known as machine unlearning , have emerged as a promising solution. Most recent machine unlearning methods [ 16 , 15 , 13 ] claim that they can effectively achieve privacy protection. In particular, it’s well-believed that exact unlearning methods [ 32 ] can surely achieve the privacy-protection goal as the target samples are exactly removed during the unlearning process, while approximate unlearning may leave residual information about the removed data [ 9 ] . While the focus remains on the unlearned samples, no unlearning method cares about the privacy of retained samples. It becomes critical to examine retrained samples when the original machine learning model is differentially private [ 1 ] . We must also maintain the level of privacy protection for the retained samples, i.e., whether the privacy budget, \epsilon , is still met. Some recent studies indicate this omission might lead to severe privacy damage. Carlini et al. [ 8 ] show that a sample’s privacy is a relative notion, meaning that removing some samples could inadvertently increase the privacy risks of other samples in the retained data. It is consistent with the understanding of anonymization by blending in the crowd – a thinning crowd reduces protection to the crowd members. No study has shown how this privacy onion effect may affect machine unlearning results. Scope of Research . We take a holistic approach to audit the effect of approximate machine unlearning algorithms on both unlearned and retained samples. Specifically, we redefine the criteria for successful machine unlearning in privacy protection 2 2 2 Machine unlearning can also be used to remove the effect of malicious or erroneous samples [ 32 ] , which may have different measuring criteria. for differentially private models: (1) the privacy risk of unlearned samples should be reduced to safe level after unlearning, and (2) the privacy risk of retained samples should remain below the privacy budget. Recent studies on auditing differentially private machine learning algorithms provide us with ideas to formally analyze and define the above two goals. The auditing mechanism uses the theorem that if a ( \epsilon,\delta )-differentially-private machine learning algorithm is correctly implemented, the most powerful membership inference attack (MIA) on every sample must satisfy \ln(\text{TPR}/\text{FPR})<\epsilon , where TPR and FPR are the true positive and false positive rates of the membership inference test, respectively [ 11 , 30 , 19 ] . With a sample-level privacy risk measure, we can formally define the new criteria for machine unlearning methods. Specifically, we define that unlearned samples need to meet an arbitrarily small privacy budget, e.g., a small value t , t=0.01 , so that a powerful MIA on an unlearned sample meets \ln(\text{TPR}/\text{FPR})<t . Meanwhile, the retained samples should still meet \ln(\text{TPR}/\text{FPR})<\epsilon . We will use these criteria to examine the existing unlearning algorithms. To make auditing effective, the MIA test must be effective, e.g., \ln(\text{TPR}/\text{FPR}) is as close to its theoretical upper bound as possible. So far, the LiRA method is considered the best MIA method that can measure an individual sample’s \ln(\text{TPR}/\text{FPR}) more accurately than other methods [ 7 , 33 ] . However, a major challenge in applying the LiRA method is its high computational cost. Online-LiRA requires 4.8 GPU hours per sample to generate the TPR/FPR measure. While offline-LiRA is more efficient due to its one-sided hypothesis testing, its performance is also reduced. In this paper, we introduce a more efficient augmentation-based likelihood ratio attack (A-LiRA), which achieves comparable quality to online-LiRA with an 88.3% reduction in time cost. A-LiRA is inspired by [ 24 ] . Instead of training multiple shadow models as in online-LiRA, A-LiRA approximates the two probability distributions – when the model is trained with or without the target sample – using augmented data and with the assumption of normal distributions for in- and out-training measures, separately. This significantly reduces the time cost while providing a good approximation of the probability distributions, making A-LiRA both effective and efficient. With A-LiRA, we have conducted extensive evaluations to show that many recent approximate machine unlearning methods do not satisfactorily protect the privacy of unlearned samples and, in some cases, increase the privacy risk of retained samples. Thus, new unlearning methods are desired to meet the proposed privacy measures. In summary, our contributions are as follows: • We re-formulate the criteria for privacy protection of machine unlearning methods for differentially private models. • We introduce a novel augmentation-based likelihood-ratio attack to estimate the privacy risk of samples, which enables efficient auditing of unlearning algorithms. • We show that most approximate machine unlearning methods do not satisfactorily protect privacy for both the unlearned and retained samples. The remaining sections are organized as follows: Related works (Section II ), preliminaries (Section III ), threat model (Section IV ), holistic auditing machine unlearning (Section V ), and experiments (Section VI ). II Related Works Machine unlearning. Machine unlearning enforces the “right to be forgotten" and is categorized into exact and approximate methods. Exact unlearning retrains models without the forgotten data. A notable example is SISA by [ 4 ] , which partitions training data into