Redirection for Erasing Memory (REM): Towards a universal unlearning method for corrupted data

Stefan Schoepf, Michael C. Mozer, Nicole Mitchell, Alexandra Brintrup, George Kaissis, Peter Kairouz, Eleni Triantafillou · 2025

Read Redirection for Erasing Memory (REM): Towards a universal unlearning method for corrupted data by Stefan Schoepf, Michael C. Mozer, Nicole Mitchell, Alexandra Brintrup, George Kaissis, Peter Kairouz, Eleni Triantafillou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17730.txt.

Introduction

Stefan Schoepf and six co-authors offer a shared map for teaching AI to forget bad data in a 2025 paper. Vision models can soak up corrupt training images, from plain errors to planted flaws. Past forgetting tools each fit one kind of task and fail past it, which made fair tests hard. The team sorts tasks by how much corrupt data is known and how alike it is.

Their method, called REM, steers corrupt inputs to fresh neurons added at forget time. Those neurons are then cut or switched off, which mutes the corrupt sway. Across the whole task map their method held strong. Past top tools broke down once they left the patch they were built for.

Catalog record

CREATORStefan Schoepf, Michael C. Mozer, Nicole Mitchell, Alexandra Brintrup, George Kaissis, Peter Kairouz, Eleni Triantafillou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17730

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Redirection for Erasing Memory (REM): Towards a universal unlearning method for corrupted data Stefan Schoepf, Michael C. Mozer, Nicole Mitchell, Alexandra Brintrup, George Kaissis, Peter Kairouz, Eleni Triantafillou Abstract Machine unlearning is studied for a multitude of tasks, but specialization of unlearning methods to particular tasks has made their systematic comparison challenging. To address this issue, we propose a conceptual space to characterize diverse corrupted data unlearning tasks in vision classifiers. This space is described by two dimensions, the discovery rate (the fraction of the corrupted data that are known at unlearning time) and the statistical regularity of the corrupted data (from random exemplars to shared concepts). Methods proposed previously have been targeted at portions of this space and—we show—fail predictably outside these regions. We propose a novel method, Redirection for Erasing Memory (REM) , whose key feature is that corrupted data are redirected to dedicated neurons introduced at unlearning time and then discarded or deactivated to suppress the influence of corrupted data. REM performs strongly across the space of tasks, in contrast to prior SOTA methods that fail outside the regions for which they were designed. 1 Introduction Unlearning is the problem of removing the effect of a subset of training data from a trained model [ 21 ] . In this work, we consider a scenario where after having already trained a model on a dataset, we discover that a subset of the training data was accidentally mislabelled, low quality, or manipulated by an attacker, causing the model to make mistakes or exhibit unwanted behaviours. The goal of unlearning (UL) is to post-process the trained model to efficiently remove (the effect of) that corrupted data in order to restore the correct predictions and behaviours. While the problem of unlearning has attracted significant attention [ 29 , 12 , 9 ] , and a plethora of methods have been proposed, there is still a lack of scientific understanding of the behaviours of these methods on different types of UL tasks, with only early work in this direction [ 32 , 9 ] . This lack of a systematic understanding of when different methods fail or succeed is a fundamental blind spot as it hinders research progress and can lead to unpredictable failure in practice as shown in this work. To address this, our first contribution is a taxonomy of tasks for unlearning corrupted data shown in Fig. 1 . Our taxonomy is based on the identification of two dimensions along which the behaviours of unlearning algorithms differ substantially. The first is the discovery rate , the fraction of corrupted data that have been identified and can be utilized by the unlearning algorithm to remove the effect of corruptions. Goel et. al [ 9 ] previously studied the effect of discovery rates on unlearning performance, finding that performance of algorithms developed for the full discovery case drops off suddenly as the discovery rate is lowered continuously. We take this study a step further by identifying a second dimension that affects performance significantly, both on its own and through interaction with the discovery rate. The second dimension is the statistical regularity of the corrupted data, tracing a spectrum from “spurious” corruptions (such as random mislabeling), towards structured corruptions that systematically affect related data points (such as a poison trigger that redirects all images on which it appears to a pre-specified class label). Our key finding is that, in our 2D space of tasks, different state-of-the-art (SOTA) methods succeed in different slices, but each fails predictably and catastrophically everywhere else. These methods are therefore risky to use in practice when the task specification is not known. # (a) (b) (c) (d) (e) (f) (g) Figure 1 : We present a new taxonomy of unlearning tasks in terms of two dimensions: the regularity and the discovery rate of the corrupted data we wish to unlearn. The highlighted areas in (a) show tasks studied in prior work. Subplots (b-e) show an aggregate metric of unlearning performance (see Section 2 ) of different methods for different discovery rates >0% (x-axis) and regularities (y-axis), instantiated via the benchmarks of [ 9 ] . Better performance is shown in darker color. Prior methods succeed in only slices of this 2D space, mainly failing along the regularity axis. To address the need for a universal method for unlearning corrupted data that covers the 2D space, we present our second contribution: Redirection for Erasing Memory (REM) , a new unlearning algorithm that is the first to perform strongly across our task space. REM employs a novel mechanism, inspired by the memorization mitigation method Example-Tied Dropout (ETD) [ 18 ] , to redirect the corruption to a dedicated part of the network (newly initialized capacity added to the model by REM) that can then be conveniently dropped off or deactivated to unlearn even undiscovered corrupted data. We benchmark REM against SOTA unlearning methods as well as ETD on low, medium, and high regularity unlearning tasks for discovery rates ranging from 10% to 100%. Experimental results on CIFAR10 [ 16 ] and Street View House Numbers (SVHN) [ 20 ] with ResNet-9 [ 13 ] and Vision Transformer (ViT) [ 3 ] as well as different optimizers show that REM is the only method to perform well or equally well across the entire 2D space. 2 Problem Definition and Background Machine unlearning applications range from privacy protection of individuals [ 19 , 26 , 12 ] to correcting errors in the model due to corrupted training data [ 8 , 17 , 9 ] , each of which has different goals and evaluation metrics. In this work, we focus on unlearning corrupted data for classification tasks. Practical examples of corrupted data can range from simple data entry errors [ 27 , 24 ] to large scale data poisoning attacks [ 1 , 31 ] . Problem definition. Let \mathcal{D}_{\text{tr}} denote a training dataset, and \mathcal{D}_{\text{test}} a held-out dataset of the same distribution. Further, let \mathcal{A} denote a training algorithm, and \theta=\mathcal{A}(\mathcal{D}) the weights of a neural network that are learned by applying \mathcal{A} on \mathcal{D} . We consider scenarios where some training data \mathcal{D}_{\text{c}}\subset\mathcal{D}_{\text{tr}} is corrupted. Let \mathcal{D}_{\text{f}}\subseteq\mathcal{D}_{\text{c}} denote the forget set comprised of corrupted data discovered after training. We refer to the rest of the training data, \mathcal{D}_{\text{tr}}\setminus\mathcal{D}_{\text{f}}=\mathcal{D}_{\text{r}} as the retain set . This formulation reduces to the most commonly-studied variation of unlearning when \mathcal{D}_{\text{f}}=\mathcal{D}_{\text{c}} ; i.e. the full set of data that we wish to unlearn is provided in the forget set. In that case, a straightforward but computationally expensive solution is to retrain from scratch to obtain a new model \theta_{r}=\mathcal{A}(\mathcal{D}_{\text{r}}) that is not affected by corrupted data. For “partial discovery”, defined as \mathcal{D}_{\text{f}}\subset\mathcal{D}_{\text{c}} , \theta_{r} is not a solution since undiscovered corrupted data \mathcal{D}_{\text{c}}\setminus\mathcal{D}_{\text{f}}=\mathcal{D}_{\text{undiscovered}} is part of \mathcal{D}_{\text{r}} , so re-training on it does not eliminate the corruptions [ 9 ] . The problem we are interested in is to design a ( \mathcal{A} , \mathcal{U} ) pair of a learning algorithm \mathcal{A} and post-processing (unlearning) algorithm \mathcal{U} such that, when a forget set \mathcal{D}_{\text{f}} is identified, \mathcal{U} can efficiently post-process the trained model \theta_{o}=\mathcal{A}(\mathcal{D}_{\text{tr}}) to yield an unlearned model \theta_{u}=\mathcal{U}(\theta_{o},\mathcal{D}_{\text{f}}) that does not suffer from failure

Read the complete text