ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song, Jinsong Su · 2026
Read ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models by Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song, Jinsong Su (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18879.txt.
Introduction
Yujie Lin and four colleagues treat machine unlearning as careful remapping of knowledge, not retraining or harsh fine-tuning. Their ZeroUnlearn method maps sensitive inputs onto a neutral target and wipes their old forms. A closed-form update rule keeps the new forms apart, and a gradient-based variant handles many samples at once. Tests show it beats prior methods at removing chosen knowledge while keeping the model's broad skills intact.
Catalog record
| CREATOR | Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song, Jinsong Su |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75382 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18879
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song, Jinsong Su Abstract Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach can outperform existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn . † † affiliationnotice: Equal contribution 1 Introduction Recently, large language models (LLMs) ( 10 ; 41 ; 1 ) have demonstrated remarkable performance across a wide range of information-intensive tasks. Since these models are often trained on extensive web corpora, they inevitably acquire and retain biased ( 37 ; 21 ; 20 ; 22 ; 33 ) , private ( 6 ; 31 ) , or outdated information ( 30 ; 39 ; 8 ) . Thus, the ability to selectively remove specific knowledge, known as machine unlearning ( 4 ) , has become a critical requirement for the responsible deployment of LLMs, particularly in scenarios demanding compliance with privacy regulations, content moderation, or factual updates. Existing approaches to unlearning in LLMs are often data-driven retraining ones, which can be mainly categorized into two primary paradigms ( 42 ; 3 ) . The first represents the naive yet exact solution: retraining the model from scratch on the remaining dataset after excluding the specific knowledge to be forgotten ( 42 ) . However, given the huge parameter scale of modern LLMs and the magnitude of pretraining corpora, the computational cost of full retraining is typically prohibitive, rendering it practically infeasible for real-world applications. The second paradigm therefore focuses on efficient fine-tuning, typically by applying penalty-based objectives (e.g., gradient ascent) directly on the forget set ( 16 ; 18 ) . While computationally more feasible, this aggressive optimization often leads to undesirable side effects, such as the unintended erosion of semantically related yet benign knowledge (neighborhood knowledge) and the degradation of the model’s core linguistic capabilities. Although subsequent studies have attempted to mitigate these issues through various regularization techniques or preservation constraints ( 43 ) , achieving an effective balance among unlearning efficacy, protection of related knowledge, and preservation of general model utility continues to pose a significant and unresolved challenge. In contrast to these traditional optimization paradigms, knowledge editing ( 28 ; 26 ; 27 ; 9 ) offers a more precise alternative. It operates by selectively modifying only a specific subset of parameters to update the model’s factual knowledge. This targeted mechanism motivates a novel hypothesis: is it possible to repurpose knowledge editing to achieve unlearning by re-mapping the targeted knowledge to a predefined safe state? Specifically, rather than destructively perturbing the model weights, we propose to overwrite sensitive information that could trigger harmful generations by assigning it a new label. Consequently, when encountering such input, the edited model will be directed to produce a neutral token such as “ <EOS> ”. To this end, we introduce ZeroUnlearn , a framework specifically designed for the few-shot knowledge unlearning. Distinct from conventional knowledge editing techniques that primarily focus on establishing a new input–output mapping, ZeroUnlearn enforces a dual objective: it not only redirects sensitive inputs to a designated target token but also explicitly minimizes the representational similarity between the updated state and the original knowledge. More concretely, we ensure that the unlearning process fundamentally orthogonalizes the edited representations with respect to their original sensitive embeddings, thereby achieving more complete erasure. To achieve this, we devise a novel multiplicative knowledge editing framework and mathematically derive a closed-form solution for the optimal transformation matrix. Furthermore, we extend our framework to multi-sample unlearning by introducing ZeroUnlearn-GD, a gradient-based variant that surpasses existing editing baselines in unlearning efficacy. In summary, our main contributions are as follows: • We propose ZeroUnlearn, a pioneering framework that reframes machine unlearning as a precise knowledge remapping task through a novel multiplicative parameter update mechanism. By projecting sensitive inputs into a null space orthogonal to their original representations, our framework ensures thorough knowledge removal while preserving the model’s general utility. • We provide a theoretical derivation for the unlearning objective, yielding a closed-form solution that enables efficient one-step optimization tailored to few-shot scenarios. Additionally, we extend this formulation to multi-sample settings via ZeroUnlearn-GD, a gradient-based variant designed to handle batch unlearning. • We conduct experiments across widely-used models and benchmarks, demonstrating that ZeroUnlearn and its variant significantly outperform baselines while maintaining a favorable balance between unlearning efficacy and general model utility. 2 Related work Knowledge Editing aims to modify specific factual knowledge within LLMs with high precision and locality. One line of methods utilizes external memory or auxiliary modules to intercept and override the model’s original predictions for targeted queries, effectively “patching” the model without altering its core weights ( 29 ; 15 ; 12 ) . Another line of research focuses on direct parameter optimization or weight manipulation. These methods typically identify specific layers responsible for storing particular knowledge and apply closed-form updates to modify factual associations ( 26 ; 27 ) . Model Unlearning seeks to comply with data-protection regulations by efficiently removing the influence of specific training samples without costly retraining procedures ( 11 ; 4 ; 32 ) . A prominent line of work formulates unlearning as an optimization problem, often applying gradient ascent on unlearning samples to suppress undesired outputs or behaviors ( 16 ; 42 ; 25 ) . Another approach treats unlearning as a supervised fine-tuning task by relabeling or rewriting the target outputs for data to be forgotten ( 8 ; 17 ; 3 ) . Through gradient descent toward alternative or neutral responses, these methods aim to overwrite unwanted knowledge while preserving the model’s overall utility. 3 Background 3.1 Unlearning for Large Language Models In the context of LLMs, we define the unlearning task as the targeted removal of specific factual associations or sensitive data. Let \mathcal{D}_{f} = \{(x_{i},y_{i})\}_{i=1}^{|\mathcal{D}_{f}|} denote the forget set , which contains information that must be neutralized due to privacy, safety, or legal requirements. Given a pre-trained model f_{\theta} parameterized by \theta , the objective of machine unlearning is to derive updated parameters \theta^{\prime} such that f_{\theta^{\prime}} no longer exhibits knowledge of \mathcal{D}_{f} . Unlike traditional retraining-based paradigms, we focus on a data-efficient setting where only the forget set \mathcal{D}_{f} is available during the unlearning process. Formally, an unlearning algorithm \mathcal{U} serves as a transformation: \theta^{\prime}=\mathcal{U}(\theta,\mathcal{D}_{f}). (1) This update is governed by two primary desiderata: (i) Forget Efficacy: The influence of \mathcal{D}_{f} on the model’s output must be effectively neutralized. This is typically achieved by re-mapping sensitive inputs to non-informative targets (e.g.,