The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples
Hsiang Hsu, Pradeep Niroula, Zichang He, Ivan Brugere, Freddy Lecue, Chun-Fu Chen · 2026
Read The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples by Hsiang Hsu, Pradeep Niroula, Zichang He, Ivan Brugere, Freddy Lecue, Chun-Fu Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22359.txt.
Introduction
Hsiang Hsu and a large team show that erased data can still haunt a model when inputs are tweaked. Machine unlearning aims to strip one user's data from a trained model without a full retrain. Their 2026 preprint notes that past proofs of erasure use clean inputs only.
When inputs get small, hostile tweaks, the erased trace can show through. They call this leftover trace residual knowledge. The gap matters for privacy law and for user trust. Their work sets a sterner test for claims of true erasure. Builders will need guards that hold under attack.
Catalog record
| CREATOR | Hsiang Hsu, Pradeep Niroula, Zichang He, Ivan Brugere, Freddy Lecue, Chun-Fu Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 132 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples Hsiang Hsu, Pradeep Niroula, Zichang He, Ivan Brugere, Freddy Lecue, Chun-Fu Chen Abstract Machine unlearning offers a practical alternative to avoid full model re-training by approximately removing the influence of specific user data. While existing methods certify unlearning via statistical indistinguishability from re-trained models, these guarantees do not naturally extend to model outputs when inputs are adversarially perturbed. In particular, slight perturbations of forget samples may still be correctly recognized by the unlearned model—even when a re-trained model fails to do so—revealing a novel privacy risk: information about the forget samples may persist in their local neighborhood. In this work, we formalize this vulnerability as residual knowledge and show that it is inevitable in high-dimensional settings. To mitigate this risk, we propose a fine-tuning strategy, named RURK, that penalizes the model’s ability to re-recognize perturbed forget samples. Experiments on vision benchmarks with deep neural networks demonstrate that residual knowledge is prevalent across existing unlearning methods and that our approach effectively prevents residual knowledge. 1 Introduction The widespread use of user data in training machine learning (ML) models has raised significant privacy concerns, particularly when user data is memorized by models such as deep neural networks, and can later be extracted or reconstructed ( 8 ; 45 ) . This memorization violates regulations such as the “Right to be Forgotten” in EU’s GDPR ( 69 ) , which mandates that, upon a user’s removal request, an ML service provider must not only delete the user data from databases but also ensure its removal from the ML models themselves ( 63 ) . As a result, simply deleting the data is often insufficient; instead, re-training the model from scratch without the specific user data is necessary. However, this re-training process is computationally expensive and impractical in real-time or large-scale settings ( 20 ) . To address this challenge, machine unlearning has been proposed as a more scalable alternative that approximately removes the influence of the specific data (referred to as forget samples) from a pre-trained model, as a substitute for avoiding full re-training ( 7 ; 4 ) . This approach, known as approximate unlearning , is often certified through statistical indistinguishability between the unlearned model and one re-trained without the forget samples ( 28 ; 10 ) . Theoretical formulations of approximate unlearning (cf. § 2 ) suggest that an unlearned model should behave similarly to a re-trained model on forget samples, for example, by producing similar predictions or classification accuracy. However, these guarantees typically apply only to the original samples and do not extend to their proximities, especially under imperceptible adversarial perturbations. In complex hypothesis spaces, even minor input changes can cause the unlearned and re-trained models to disagree in their predictions, creating an additional layer of privacy risk. A particularly concerning case is when a slightly perturbed forget sample is still correctly classified by the unlearned model but not by the re-trained one. This reveals the presence of residual knowledge —latent traces of the forget samples that still persist in the unlearned model (cf. Figure 1 ). Residual knowledge is a prevalent issue. For instance, on the CIFAR-10 dataset, when subjected to a small perturbation norm ( \approx 0.03 ), over 7% of the forget samples still exhibit residual knowledge (cf. Appendix C.4 ). In this paper, we study how adversarially perturbed inputs affect the indistinguishability between unlearned and re-trained models in the classification setting. In § 3 , we show that adversarial examples can reliably distinguish between the two models, even under certified approximate unlearning. We formalize this observation by demonstrating that adversarial attacks can induce probabilistically distinguishable outputs. Further, using geometric probability ( 66 ) , we prove that such disagreement is inevitable in high-dimensional input spaces. These findings underscore the need to explicitly incorporate robustness against such local disagreement into unlearning frameworks. To capture this risk, § 4 introduces the notion of residual knowledge, which quantifies the likelihood that an unlearned model retains predictive traces of forget samples under perturbation. As a more tractable proxy for disagreement, residual knowledge exposes a vulnerability not addressed by current certification methods. To mitigate this, we propose RURK , a fine-tuning strategy for R obust U nlearning that suppresses R esidual K nowledge while maintaining accuracy on the rest of the samples. Our approach identifies and penalizes perturbed inputs that the unlearned model still classifies correctly, thus enhancing robustness against residual knowledge. We empirically validate the existence of such local disagreement and residual knowledge across multiple unlearning algorithms in § 5 . We further demonstrate that our fine-tuning strategy effectively reduces residual knowledge on standard vision datasets using deep neural networks. To the best of our knowledge, this is the first work to uncover this novel privacy risk and provide a scalable solution to address it. We conclude in § 6 with a discussion of limitations and future research directions. Omitted proofs, additional explanations and discussions, details on experiment setups and training, and additional experiments are included in Appendices A , B and C , respectively. Figure 1 : The re-trained (brown) and unlearned (green) models are statistically similar but may have slightly different decision boundaries ( left ), leading to disagreements on forget samples ( right ). Checkmarks and crosses on the images indicate correct and incorrect predictions from the re-trained model (top) and the unlearned model (bottom), respectively. Ideally—as shown with forget sample 1—both models should behave consistently across the original and all perturbed inputs. Residual knowledge in machine unlearning is illustrated by comparing prediction correctness: for forget sample 2, both models agree on the original sample, but the unlearned model correctly predicts more of its perturbed variants. see Appendix B.5 for experimental details. 2 Background and related work Let {\mathcal{S}}\triangleq\{{\mathbf{s}}_{i}=({\mathbf{x}}_{i},y_{i})\}_{i=1}^{n} be a training dataset with {\mathbf{x}}_{i}\in{\mathcal{X}}\subseteq\mathbb{R}^{d} and y_{i}\in{\mathcal{Y}} . The hypothesis space {\mathcal{H}}\triangleq\{h_{\mathbf{w}}:{\mathcal{X}}\to{\mathcal{Y}};{\mathbf{w}}\in{\mathcal{W}}\} consists of models parameterized by {\mathbf{w}} . We use h and {\mathbf{w}} interchangeably to refer to a model in {\mathcal{H}} . Let \ell:{\mathcal{W}}\times{\mathcal{S}}\to{\mathbf{R}}^{+} be the loss function, and define the empirical risk as L({\mathbf{w}},{\mathcal{S}})=\frac{1}{n}\sum_{i=1}^{n}\ell({\mathbf{w}},{\mathbf{s}}_{i}) . A (randomized) learning algorithm A:{\mathcal{S}}\to{\mathcal{H}} , such as Stochastic Gradient Descent (SGD), returns a model minimizing L({\mathbf{w}},{\mathcal{S}}) and induces a distribution over {\mathcal{H}} . We denote the \ell_{p} -norm by \|{\mathbf{z}}\|_{p} , and the \ell_{p} ball of radius \tau centered at {\mathbf{x}} by {\mathcal{B}}_{p}({\mathbf{x}},\tau)\triangleq\{{\mathbf{z}}\in{\mathcal{X}}:\|{\mathbf{z}}-{\mathbf{x}}\|_{p}\leq\tau\} . Also, let \mathbbm{1}(\cdot) be the indicator function, \mathsf{surf}({\mathcal{Z}}) the surface area of a set {\mathcal{Z}} in \mathbb{R}^{d} , and {\mathcal{O}}(\cdot) the big-O notation. 2.1 Machine unlearning The forget set {\mathcal{S}}_{f}\subseteq{\mathcal{S}} contains training samples to be removed. Machine unlearning is modeled as a randomized mechanism M(A({\mathcal{S}}),{\mathcal{S}},{\mathcal{S}}_{f}) that removes the influence of {\mathcal{S}}_{f} from a pre-trained model A({\mathcal{S}}) ( 73 ) . Like the learning algorithm itself, the output of the unlearning mechanism can be regarded as a random variable. A simple approach adds Gaussian noise to the model weights: M({\mathbf{w}},{\mathcal{S}},{\mathcal{S}}_{f})={\mathbf{w}}+\sigma{\mathbf{n}} , with {\mathbf{n}}\sim\mathcal{N}(0,{\mathbf{I}}_{|{\mathbf{w}}|}) ( 22 ) . However, as \sigma increases, the model becomes increasingly independent of the training data, potentially degrading performance. To preserve utility, unlearning should retain accuracy on the retain set {\mathcal{S}}_{r}\triangleq{\mathcal{S}}\setminus{\mathcal{S}}_{f} . A naïve yet effective strategy is to re-train the model from scratch on the retain set, i.e., M(A({\mathcal{S}}),{\mathcal{S}},{\mathcal{S}}_{f})=A({\mathcal{S}}_{r})