CURaTE: Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge

Seyun Bae, Seokhan Lee, Eunho Yang · 2026

Read CURaTE: Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge by Seyun Bae, Seokhan Lee, Eunho Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14644.txt.

Introduction

Bae, Lee, and Yang propose CURaTE, a method for continual real-time unlearning that preserves language-model knowledge. Existing methods ignore continuous, immediate action, so utility degrades as updates pile up and sensitive information stays exposed. CURaTE trains a sentence embedding model to draw sharp decision boundaries for spotting prompts matching forget requests. Prompt similarity to forget requests decides between answering and refusing. By never modifying model parameters, CURaTE forgets better than existing methods, keeps knowledge nearly perfect over any number of updates, and alone manages continual real-time unlearning.

Catalog record

CREATORSeyun Bae, Seokhan Lee, Eunho Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70270
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14644

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CURaTE: Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge Seyun Bae, Seokhan Lee, Eunho Yang Abstract The inability to filter out in advance all potentially problematic data from the pre-training of large language models has given rise to the need for methods for unlearning specific pieces of knowledge after training. Existing techniques overlook the need for continuous and immediate action, causing them to suffer from degraded utility as updates accumulate and protracted exposure of sensitive information. To address these issues, we propose C ontinual U nlearning in R e a l T ime with E nsured Preservation of LLM Knowledge ( CURaTE ). Our method begins by training a sentence embedding model on a dataset designed to enable the formation of sharp decision boundaries for determining whether a given input prompt corresponds to any stored forget requests. The similarity of a given input to the forget requests is then used to determine whether to answer or return a refusal response. We show that even with such a simple approach, not only does CURaTE achieve more effective forgetting than existing methods, but by avoiding modification of the language model parameters, it also maintains near perfect knowledge preservation over any number of updates and is the only method capable of continual unlearning in real-time. † † footnotetext: Code is available at https://github.com/bsu1313/CURaTE . 1 Introduction The vast amount of text that is indiscriminately scraped from corpora on the web in order to train large language models makes it infeasible to effectively filter out in advance all the data that could later become the source of contention, such as copyrighted content, sensitive or dangerous information, and false information. This has given rise to a plethora of techniques for modifying a pre-trained large language model such that it unlearns specific problematic pieces of information that were contained in its training data ( 7 ; 16 ) . The problem with most current approaches is that they start by defining the task of unlearning narrowly to include only methods that involve directly modifying the weights of the LLM ( 35 ; 8 ) . This inevitably results in severe performance degradation due to catastrophic forgetting, a problem that only becomes worse with every new update in the continual setting. In addition, current methods do not take into account the time required to carry out the unlearning process. Methods that are inefficient in this regard risk exposing sensitive information while the process is underway. The solution proposed in this paper, is to widen the scope of possible techniques by introducing the concept of behavioral unlearning which encompasses a broader class of methods and subsumes the traditional methods—which we term parametric unlearning . For behavioral unlearning, the goal is not so much to erase the knowledge from the language model per se , but rather to ensure the language model does not output information that has been flagged to be problematic—and to do so in a timely manner, while ensuring that no other knowledge is lost in the process. To solve this more usefully defined problem , we introduce C ontinual U nlearning in R e a l T ime with E nsured Preservation of LLM Knowledge ( CURaTE ). Prior to LLM deployment, CURaTE trains an unlearning sentence embedding model on a synthetically generated dataset with hard negatives designed to enable fine-grained classification between user queries related to the forget set , and those that are unrelated. After the LLM is deployed, CURaTE continuously adds the embeddings of any received forget requests to its embedding database in real-time and compares them with the embedding of the current user query. Then based on this comparison, we decide whether the LLM should provide a response to the user query or refuse. Importantly, since the unlearning embedder does not require any additional training post-deployment—and in particular, does not need to use either the forget set or the retain set for training, the entire process achieves significantly faster unlearning compared to prior approaches. Moreover, because the weights of the LLM remain unmodified, CURaTE allows for near perfect utility preservation. As a result, not only does our method substantially outperform all other unlearning methods in the continual setting (which is the setting most relevant to real world applications), it is the first method we are aware of that is capable of processing ongoing forget requests in real-time with minimal degradation of model performance as requests accumulate over time. In summary, the contributions of our work are as follows: • We introduce CURaTE , an unlearning framework that entails virtually no overhead for processing new forget requests and thus constitutes the first unlearning method capable of handling continual, sequential forget requests in real-time. • Through experiments across multiple benchmarks , we demonstrate that by leaving the weights of the LLM unmodified, CURaTE is able to largely circumvent the catastrophic forgetting problem faced by existing methods and achieve near perfect preservation of LLM knowledge, even after processing a long succession of continual forget requests. • We demonstrate superiority over prior state-of-the-art (SOTA) unlearning methods in additional aspects such as the ability of our method to generalize to any unlearning task after training on a single dataset (whereas existing methods typically require retraining on every new forget and retain set ), and robustness to paraphrased variants of sentences in the forget set . 2 Related Work Conventional Unlearning. Methods that only use the forget set for training are called Gradient Ascent (GA) ( 7 ) . These methods train the target LLM to minimize a loss on the forget set defined as the positive log likelihood of the text in the forget set , thereby minimizing the likelihood of generating the information contained in the forget set . Other methods add to this loss by including a term for the negative log likelihood of the text in the retain set , which acts as a regularizer forcing the LLM to not only forget the information in the forget set but to also explicitly remember the information in the retain set . These methods are known as Gradient Difference (GradDiff) ( 14 ) . A third approach, called Preference Optimization (PO) ( 20 ) , uses a loss that encompasses terms for both the forget set and the retain set , but instead of using the positive log likelihood on the forget set , it uses the negative log likelihood on alternate refusal responses to the questions in the forget set . Negative Preference Optimization (NPO) ( 35 ) uses the loss from Direct Preference Optimization (DPO) ( 25 ) but with only negative examples (instead of pairs of positive and negative examples). More recent work includes SOUL ( 8 ) , which is not of itself a distinct unlearning method, but rather an improvement that adds second-order optimization to existing methods. These methods tend to have weak performance on knowledge preservation metrics as modifying weights inevitably results in catastrophic forgetting. Weight Preserving Unlearning. Existing approaches that avoid modifying LLM weights include In-Context Unlearning (ICUL) ( 24 ) which adds data points from the forget set with perturbed labels as in-context examples to the LLM prompt, and guardrail methods ( 31 )

Read the complete text