Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy
Jie Ren, Zhenwei Dai, Xianfeng Tang, Yue Xing, Shenglai Zeng, Hui Liu, Jingying Zeng, Qiankun Peng, Samarth Varshney, Suhang Wang, Qi He, Charu C. Aggarwal, Hui Liu · 2025
Read Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy by Jie Ren, Zhenwei Dai, Xianfeng Tang, Yue Xing, Shenglai Zeng, Hui Liu, Jingying Zeng, Qiankun Peng, Samarth Varshney, Suhang Wang, Qi He, Charu C. Aggarwal, Hui Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00359.txt.
Introduction
Jie Ren and a team of co-authors wrote this 2025 paper on a hidden risk in teaching language models to forget. Unlearning aims to strip out harmful or private training data without rebuilding the model. The team shows a crafted forget request can quietly wreck the model for normal users.
The trick pads the forget data with common everyday words. Models tend to tie such words to the act of forgetting itself, so later, when a normal prompt holds those words, the model plays dumb or rambles. The fix adds a scope term to the training goal, teaching the model to keep the forgetting local. The authors frame the pair as an attack and a guard that any forget system should now reckon with.
Catalog record
| CREATOR | Jie Ren, Zhenwei Dai, Xianfeng Tang, Yue Xing, Shenglai Zeng, Hui Liu, Jingying Zeng, Qiankun Peng, Samarth Varshney, Suhang Wang, Qi He, Charu C. Aggarwal, Hui Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy Jie Ren, Zhenwei Dai, Xianfeng Tang, Yue Xing, Shenglai Zeng, Hui Liu, Jingying Zeng, Qiankun Peng, Samarth Varshney, Suhang Wang, Qi He, Charu C. Aggarwal, Hui Liu Abstract Although Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of tasks, growing concerns have emerged over the misuse of sensitive, copyrighted, or harmful data during training. To address these concerns, unlearning techniques have been developed to remove the influence of specific data without retraining from scratch. However, this paper reveals a critical vulnerability in fine-tuning-based unlearning: a malicious user can craft a manipulated forgetting request that stealthily degrades the model’s utility for benign users. We demonstrate this risk through a red-teaming Stealthy Attack (SA) , which is inspired by two key limitations of existing unlearning—the inability to constrain the scope of unlearning effect and the failure to distinguish benign tokens from unlearning signals. Prior work has shown that unlearned models tend to memorize forgetting data as unlearning signals, and respond with hallucinations or feigned ignorance when unlearning signals appear in the input. By subtly increasing the presence of common benign tokens in the forgetting data, SA enhances the connection between benign tokens and unlearning signals. As a result, when normal users include such tokens in their prompts, the model exhibits unlearning behaviors, leading to unintended utility degradation. To address this vulnerability, we propose Scope-aware Unlearning (SU) , a lightweight enhancement that introduces a scope term into the unlearning objective, encouraging the model to localize the forgetting effect. Our method requires no additional data processing, integrates seamlessly with existing fine-tuning frameworks, and significantly improves robustness against SA. Extensive experiments validate the effectiveness of both SA and SU. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, such as question answering kwiatkowski2019natural , machine translation stahlberg2020neural , text summarization el2021automatic , and dialogue generation li2016deep . This rapid advancement has been largely driven by training on massive datasets. However, growing concerns have been raised about the potential misuse of undesirable data in training, such as copyrighted materials, privacy-sensitive information, or harmful content hacker2023regulating ; lucchi2024chatgpt . For instance, New York Times sued OpenAI and Microsoft for using its articles for training 1 1 1 https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html . Under the General Data Protection Regulation (GDPR) voigt2017eu , the data is protected, and data owners—like The New York Times—retain the “right to be forgotten” rosen2011right . To address these concerns, unlearning techniques have been proposed to remove the influence of specific data from large language models (LLMs) without retraining them from scratch maini2024tofu ; yao2023large ; liu2022continual ; zhang2024negative ; fan2024simplicity ; ren2024copyright . Specifically, model providers who develop and deploy LLM can offer unlearning services that accept user-submitted data removal requests. Upon receiving a set of data to be forgotten, the provider applies an unlearning method to update the model, which is then expected to avoid generating content that reflects the removed information. However, despite the advantages, there is an overlooked problem in offering such services: Is it safe to blindly trust and unlearn user-submitted data without concern? A malicious user may submit a disguised forgetting dataset in an unlearning request to intentionally degrade model utility—particularly when the unlearned model is later provided to normal users. In this paper, by proposing a Stealthy Attack (SA) , we demonstrate that fine-tuning-based LLM unlearning—the most prominent and widely adopted category—suffers from this critical vulnerability. Stealthy Attack (SA) is motivated by two fundamental limitations in current unlearning mechanisms. First , existing methods struggle to constrain the scope of unlearning effectiveness, as highlighted in prior studies thaker2024position ; huu2025improving . For instance, after unlearning the book “ Watermelon on the Moon ”, a model is asked: “ Who is the author of Watermelon on the Moon? And where is the Eiffel Tower? ” Although the second question is unrelated, the model fails to answer it—an unintended consequence of overgeneralization from unlearning process. Other studies attribute this issue to the fact that unlearning in LLMs does not truly erase the forgetting data deeb2024unlearning ; shi2024muse ; zhang2024catastrophic , but rather memorizes it as a signal to simulate ignorance or hallucinate ren2025general . Second , fine-tuning-based unlearning methods inadvertently treat benign tokens—those unrelated to the forgetting knowledge—as unlearning signals. This occurs because such methods cannot distinguish between tokens that encode the forgetting knowledge (referred to as target tokens) and unrelated benign tokens, as illustrated in Section 3.2 . These limitations open the door for malicious users to attack the unlearning process by increasing the model’s reliance on common, benign tokens such as “ please ” and “ then ”. When the unlearned model is later deployed to normal users—who naturally use these benign tokens—the model’s performance degrades. Stealthy Attack exploits this vulnerability by intentionally increasing the influence of the unlearning loss on selected benign tokens (referred to as benign triggers ) through subtle frequency manipulation. Experiments in Section 5 demonstrate the effectiveness of this attack, revealing a critical vulnerability in current unlearning techniques. To mitigate the above vulnerability and enhance the robustness of unlearning, we propose Scope-aware Unlearning (SU) , a simple yet effective improvement to the fine-tuning-based unlearning framework. In addition to the standard forgetting and retaining losses, we introduce a scope term that constrains the effect of unlearning to the appropriate region within the target knowledge. This is achieved by constructing scope samples that combine forgetting and retaining prompts, training the model to ignore unlearning signals when they appear in normal contexts. Our method requires no additional processing—only the addition of a single term to the unlearning objective. Experimental results show that this approach significantly improves robustness against SA. Moreover, it is model-agnostic and compatible with existing fine-tuning-based unlearning methods, making it easy to integrate into current workflows. 2 Related works Machine Unlearning. Machine Unlearning is first proposed in the domain of computer vision and classification models cao2015towards ; warnecke2021machine ; bourtoule2021machine ; kurmanji2024towards ; ren2024copyright . Recently, it has been extended into the area of LLMs to mitigate the risks associated with undesirable data eldan2023s ; yao2023large ; shi2024muse ; wang2024machine ; eldan2023s ; ji2024reversing ; thaker2024guardrail ; liu2024large . Fine-tuning-based unlearning is the most widely used category due to its straightforward motivation and promising performance thaker2024position ; deeb2024unlearning ; doshi2024does ; lynch2024eight . The existing fine-tuning objectives can be divided into two categories. The first is Gradient Ascent (GA) jang2023knowledge ; maini2024tofu and its variants yao2023large ; liu2022continual ; zhang2024negative ; fan2024simplicity ; veldanda2024llm ; cha2024towards ; liu2024towards ; feng2024fine ; bu2024unlearning ; tian2024forget . These methods fine-tune the LLM with the reversed training loss on forgetting data to negate the training impact. The second is the preference-based methods maini2024tofu ; liwmdp ; wang2024llm ; huu2024effects ; shi2024ulmr ; liu2024towards ; sinha2024unstar . These methods do not aim at removing the influence of forget data. Instead, they instruct the model to generate human-preferred response to forgetting data such as “I don’t know”. The mechanism of unlearning in LLM. Although the goal of unlearning, especially GA-based methods, is to remove the influence of forgetting data, more research works reveal that LLM unlearning is not actually unlearning or