Forgetting Similar Samples: Can Machine Unlearning Do it Better?

Heng Xu, Tianqing Zhu*, Dayong Ye, Lefeng Zhang, Le Wang, Wanlei Zhou · 2026

Read Forgetting Similar Samples: Can Machine Unlearning Do it Better? by Heng Xu, Tianqing Zhu*, Dayong Ye, Lefeng Zhang, Le Wang, Wanlei Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06938.txt.

Introduction

Heng Xu and co-authors posted a 2026 arXiv preprint. The preprint deals with forgetting similar samples. It came out on arXiv in 2026 as a PAPERS preprint. Machine unlearning, a process enabling pre-trained models to remove the influence of specific training samples, has attracted significant attention in recent years. Although extensive research has focused on developing efficient machine unlearning strategies.

We argue that these methods mainly aim at removing samples rather than removing samples’ influence on the model. Thus overlooking the fundamental definition of machine unlearning. Readers in the field get a short, focused account of the problem and the steps the authors take.

Catalog record

CREATORHeng Xu, Tianqing Zhu*, Dayong Ye, Lefeng Zhang, Le Wang, Wanlei Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06938

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Forgetting Similar Samples: Can Machine Unlearning Do it Better? Heng Xu, Tianqing Zhu*, Dayong Ye, Lefeng Zhang, Le Wang, Wanlei Zhou Abstract Machine unlearning, a process enabling pre-trained models to remove the influence of specific training samples, has attracted significant attention in recent years. Although extensive research has focused on developing efficient machine unlearning strategies, we argue that these methods mainly aim at removing samples rather than removing samples’ influence on the model, thus overlooking the fundamental definition of machine unlearning. In this paper, we first conduct a comprehensive study to evaluate the effectiveness of existing unlearning schemes when the training dataset includes many samples similar to those targeted for unlearning. Specifically, we evaluate: Do existing unlearning methods truly adhere to the original definition of machine unlearning and effectively eliminate all influence of target samples when similar samples are present in the training dataset? Our extensive experiments, conducted on four carefully constructed datasets with thorough analysis, reveal a notable gap between the expected and actual performance of most existing unlearning methods for image and language models, even for the retraining-from-scratch baseline. Additionally, we also explore potential solutions to enhance current unlearning approaches. 1 Introduction Machine unlearning refers to removing the influence of specific training samples on a machine learning model [ 29 ] . This technological advancement has recently drawn urgent attention due to several factors, including the strict enforcement of the right to be forgotten in regulations and laws [ 16 , 26 , 36 ] , escalating concerns about data privacy [ 24 , 6 , 7 , 28 ] , and the pressing need to erase harmful, malicious, and even illegal knowledge from large language models [ 20 , 1 , 27 , 19 , 41 ] . Research Gap: Since being proposed, machine unlearning has been consistently defined as the process of eliminating the complete influence of a target sample [ 4 , 2 , 6 , 25 , 28 , 11 , 12 , 29 ] . Meanwhile, in realistic scenarios, datasets often contain samples that, despite differing in expression, remain closely related to target samples, and cause similar influence to the model 1 1 1 We employ the public dataset PKU-SafeRLHF to illustrate the existence of such similar samples, with the corresponding results shown in Figure 19 . . As shown in Figure 1 , consider a binary classification model that separates the training dataset into two regions. In the magnified view, a target sample (triangle) is highlighted, surrounded by similar samples (circles) that are similar to it. These similar samples will cause a similar influence on the trained model as the target sample. Accordingly, based on the original definition, effective machine unlearning methods should go beyond removing the target sample itself to also mitigating influence from other similar samples. Figure 1 : Illustration of samples near target sample. However, most existing unlearning methods can be categorized into two lines of research, with limited attention paid to the aforementioned case. The first line assumes that the target samples are independent and do not share any influence with other training samples [ 4 , 24 , 28 ] . These methods typically overlook the presence of similar samples and focus only on the target sample during the unlearning process. The second line of work has begun to explore the influence of duplicate samples and adversarial test embeddings, but it remains preliminary [ 22 , 33 , 37 ] . For example, Ye et al. [ 35 ] measured the impact of duplicated samples on unlearning using coarse-grained accuracy metrics and mainly focused on image models, but did not propose any practical or effective solutions, and left more complex models like LLMs unconsidered. Similarly, Minh et al. [ 22 ] evaluated how adversarially similar embeddings impact unlearning performance during prediction, but did not systematically investigate the influence of similar samples in the training dataset . These early studies demonstrate the importance of considering similar samples in the unlearning process. Yet, existing works offer neither a systematic analysis nor effective solutions, leaving a significant gap that our research aims to fill by thoroughly investigating these effects and proposing corresponding strategies. Why is it important to consider similar samples? These questions are not only important but also fundamental, as they directly challenge the current machine unlearning methods in real-world scenarios. If unlearning methods fail to account for similar samples, the consequences can be serious: • Incomplete unlearning : Retaining the influence of similar samples leads to incomplete unlearning, where the model continues to preserve residual influence of the target sample through its similar samples. This not only directly undermines strict privacy mandates such as GDPR and CCPA [ 16 , 26 ] , which require the complete removal of specific data influences, but also is inconsistent with the definition of machine unlearning [ 4 , 2 , 6 , 25 , 28 , 12 ] . • Model integrity: Ignoring similar samples threatens the integrity of the unlearned model. Their residual influence of similar samples can distort decision boundaries, degrade model utility, and even introduce hidden biases—ultimately making the unlearned model less trustworthy and misaligned with its intended behavior. • Potential exploitation: Overlooked similar samples create exploitable vulnerabilities. Adversaries may infer information about the target sample or introduce maliciously crafted similar samples that still persist in the unlearned model, thereby bypassing the unlearning process and potentially enabling harmful behaviors. Preliminary Experiments: We first perform a comprehensive evaluation from two key perspectives: (1). If the target sample is unlearned from model, can its similar samples also be successfully unlearned? and (2). When similar samples cannot be unlearned, could the retention of these similar samples affect the unlearning results of the target sample? To facilitate this study, we first construct four benchmark datasets: three image-based datasets and one Q&A language dataset 2 2 2 Given the challenges associated with constructing text datasets—such as sample paraphrasing and manual selection—we only use one text dataset. . Each dataset contains a different number of target samples along with their corresponding similar samples. For the image datasets, we generate similar samples by applying appropriate perturbations to the target samples. For the language dataset, we select and paraphrase target samples multiple times, standardizing those paraphrased samples to generate similar ones. To evaluate the unlearning results of both target samples and similar samples, we assess various current machine unlearning methods and introduce two more fine-grained verification methods, namely data reconstruction-based and ROUGE metrics-based [ 31 , 3 , 23 , 32 ] , instead of MIAs-based and backdoor-based sample-level verification schemes [ 29 ] . The latter two verification methods mainly assess the overall presence or absence of a complete sample in the model [ 35 ] . In contrast, our introduced methods enable a more fine-grained evaluation by capturing the influence of partial components of a sample. For the image model, we compare the recovered samples before and after the unlearning process by assessing the similarity of the pixels to the original target samples. For the language model, we use the ROUGE score to evaluate the knowledge of the model against the ground truth before and after the unlearning process. Our Findings: Based on the

Read the complete text