Similarity-Aware Machine Unlearning
Madhavan Citalamangalam Kumaran, Midhun Parakkal Unni, Vicky Kouni, Haripriya Harikumar · 2026
Read Similarity-Aware Machine Unlearning by Madhavan Citalamangalam Kumaran, Midhun Parakkal Unni, Vicky Kouni, Haripriya Harikumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00246.txt.
Introduction
Kumaran and co-authors make machine unlearning aware of similarity between data points. Unlearning removes the influence of chosen training examples from a model. It avoids retraining from scratch. Localization methods pick a subset of influential parameters to speed the work. Past picks used forget-set importance alone. That can damage retained examples that resemble the forgotten ones.
Their retain-aware method weighs parameter importance for both data groups. They also build a retain-similar evaluation set using cosine similarity in embedding space. It measures collateral damage directly. Across eleven settings on CIFAR-10 with ResNet18, their method cut collateral damage. Standard unlearning metrics improved at the same time. Retain-aware localization proved effective across those runs.
Catalog record
| CREATOR | Madhavan Citalamangalam Kumaran, Midhun Parakkal Unni, Vicky Kouni, Haripriya Harikumar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2608.00246
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Similarity-Aware Machine Unlearning Madhavan Citalamangalam Kumaran, Midhun Parakkal Unni, Vicky Kouni, Haripriya Harikumar Abstract Machine unlearning removes the influence of user-specified training examples from a trained model, avoiding the need to retrain it from scratch. Localization-based methods improve unlearning efficiency by identifying a subset of influential model parameters. However, existing approaches select parameters based solely on forget-set importance, neglecting their role in retained dataset and often causing collateral damage to semantically similar retained examples. We address this limitation with a retain-aware localization method that considers parameter importance to both forgotten and retained data. We also introduce a retain-similar evaluation set, constructed using cosine similarity in the model embedding space, to directly measure collateral damage. Across eleven experimental settings on CIFAR-10 dataset and ResNet18 model, our method consistently reduces collateral damage while improving standard unlearning metrics, demonstrating the effectiveness of retain-aware localization for similarity-aware machine unlearning. 1 Introduction Machine learning models can memorize information about individual training examples to varying degrees [ 26 ] , creating privacy risks even when the underlying training records are not directly accessible. Membership-inference attacks can reveal whether an example was used for training [ 23 ] , model-inversion attacks [ 11 ] can expose sensitive attributes [ 21 ] , and generative models can reproduce passages from their training corpora verbatim [ 6 ] . These risks are amplified when models are trained on personal or sensitive data [ 20 , 21 ] . Data-protection frameworks such as the General Data Protection Regulation [ 25 ] and California Consumer Privacy Act [ 19 ] establish rights concerning the erasure or deletion of personal data. Together, these privacy risks and data-deletion rights motivate mechanisms for removing the residual influence from models. Machine unlearning [ 5 , 2 , 27 ] addresses this problem by modifying a trained model so that its behavior approximates that of an oracle model retrained without a user-specified subset of training examples, referred to as the forget set. Retraining from scratch on the remaining data provides the best solution, but can be computationally expensive, particularly when deletion requests are frequent. Approximate unlearning [ 17 ] methods therefore seek to approximate the behavior of the retrained model at substantially lower computational cost. Localized unlearning [ 24 ] methods reduce this cost by restricting model modifications to a selected subset of parameters that are influential for the forget set. However, importance to the forget set does not imply specificity to the forget set. A parameter may be highly influential for forgotten examples while also supporting retained examples. This overlap is especially consequential when forgotten and retained examples occupy nearby regions of the learned representation space and may depend on shared features, as shown in Fig. 1 (visually similar images in red and green boxes naturally cluster closely in their embedding space). Figure 1: Intermediate feature-space visualization showing that semantically similar samples are embedded close to one another within class-specific clusters. The image to forget (red) and the image to retain (green) occupy very close neighboring regions of the feature manifold, indicating shared feature representations. Consequently, forget-only localization can modify parameters used by both sets, causing unintended degradation on representation-similar retained examples. We refer to this representational overlap as feature entanglement and to the resulting degradation as collateral damage . Because such damage may be concentrated among a small subset of retained examples, aggregate test and retain-set accuracy may fail to reveal it. To address this limitation, we propose a retain-aware localization framework. We compute criticality scores separately on the forget set and the retain set, and combine them to prioritize neurons that are important to the forget set but comparatively less important to the retain set. To measure collateral damage directly, we also construct a retain-similar evaluation set. For each forgotten example, this set contains its nearest same-class retained neighbor according to cosine similarity in the original model’s embedding space. Our contributions are as follows. • We identify collateral damage to semantically similar retained examples as a key limitation of existing localization-based machine unlearning methods and relate this phenomenon to shared learned representations. • We propose a retain-aware localization framework that incorporates parameter importance with respect to both forgotten and retained data, enabling similarity-aware unlearning. • We introduce a retain-similar evaluation protocol and demonstrate across eleven experimental settings that our approach consistently reduces collateral damage while improving standard unlearning metrics. 2 Background We introduce the notation and definitions used throughout the paper before presenting our method. Let \mathcal{D}:=\bigcup_{n\geq 1}(\mathcal{X}\times\mathcal{Y})^{n} denote the set of all datasets and \mathcal{A} a randomized training algorithm, \mathcal{A}:\mathcal{D}\rightarrow\Delta(\mathcal{H}) , where \Delta(\mathcal{H}) denotes the set of all probability distributions over the hypothesis space \mathcal{H} . The hypothesis space \mathcal{H}:=\{f_{\theta}:\mathcal{X}\to\mathcal{Y}\mid\theta\in\mathbb{R}^{d}\} is the set of all models that the learning algorithm \mathcal{A} produces, parameterized by weight vectors \theta\in\mathbb{R}^{d} . We denote by f_{\theta^{o}}\sim\mathcal{A}(D_{train}\subset\mathcal{D}) the original model parameterised by \theta^{o} when trained on the training set D_{train}\subset\mathcal{D} prior to unlearning. Let the forget set be denoted by D_{f}\subset D_{train} , consisting of the data requested for removal. The retain set is defined as, D_{r}=D_{train}\setminus D_{f} representing the remaining training data. We denote f_{\theta^{r}}\sim\mathcal{A}(D_{r}) as oracle model and \theta^{r} as the associated parameters by training from scratch only on the retain set alone. Machine unlearning is defined as the process of modifying the parameters of a model from \mathcal{A}(D_{train}) , in order to remove the influence of D_{f} without compromising performance on the retain set D_{r} . Formally, Definition 1 (Unlearning [ 3 ] ) An unlearning algorithm \mathcal{U} is said to unlearn a forget set D_{f}\subset D_{\text{train}} if the unlearned model f_{\theta^{u}}\sim\mathcal{U}(f_{\theta^{o}},D_{f},D_{r}) is indistinguishable from the retrained model f_{\theta^{r}}\sim\mathcal{A}(D_{r}) , where D_{r}=D_{\text{train}}\setminus D_{f} . Formally, the output distributions of f_{\theta^{u}} and f_{\theta^{r}} are identical on the evaluation distribution. Definition 1 characterizes the goal of machine unlearning where the unlearned model f_{\theta^{u}} should behave indistinguishably from a model retrained from scratch on the retain set D_{r} . One way to satisfy this definition is exact unlearning, where the model is retrained from scratch using only D_{r} , to produce f_{\theta^{r}}\sim\mathcal{A}(D_{r}) . Although exact unlearning directly satisfies the definition, retraining with large models and datasets can be computationally expensive. Therefore, practical approaches often employ approximate unlearning, where an unlearning algorithm \mathcal{U} modifies the original model f_{\theta^{o}} to obtain f_{\theta^{u}}\sim\mathcal{U}(f_{\theta^{o}},D_{f},D_{r}) . The objective is to have an unlearned model f_{\theta^{u}} that produces outputs closely matched to those of the retrained model f_{\theta^{r}} , while requiring significantly less computation. More precisely, machine unlearning approach aims for the following: \displaystyle t\bigl(\mathcal{U}(f_{\theta^{o}},\,D_{f},D_{r})\bigr) \displaystyle\ll t\bigl(\mathcal{A}(D_{r})\bigr), (1) \displaystyle\mathcal{U}(f_{\theta^{o}},\,D_{f},D_{r}) \displaystyle\approx\mathcal{A}(D_{r}), (2) where t(\cdot) denotes the execution time of the unlearning or training process. We use the following definition to operationalise evaluation of unlearning in line with Torkzadehmahani et al. [ 24 ] . Definition 2 (Localized Machine Unlearning [ 24 ] ) Let \theta^{o}=\{\theta_{k}\}_{k=1}^{N} denote the parameters of the original model, where N is the total number of trainable parameters. An unlearning algorithm \mathcal{U} is said to perform localized machine unlearning if, given a forget set D_{f} , it identifies and modifies only a subset of parameters \theta_{s}\subset\theta^{o},|\theta_{s}|=n<N, while leaving the remaining parameters unchanged during unlearning. The resulting unlearned model is from \mathcal{U}(f_{\theta^{o}},D_{f},D_{r}), where updates are restricted to the selected parameter subset \theta_{s}