A Comparative Study of Machine Unlearning Techniques for Image and Text Classification Models
Omar M. Safa, Mahmoud M. Abdelaziz, Mustafa Eltawy, Mohamed Mamdouh, Moamen Gharib, Salaheldin Eltenihy, Nagia M. Ghanem, and Mohamed M. Ismail · 2024
Read A Comparative Study of Machine Unlearning Techniques for Image and Text Classification Models by Omar M. Safa, Mahmoud M. Abdelaziz, Mustafa Eltawy, Mohamed Mamdouh, Moamen Gharib, Salaheldin Eltenihy, Nagia M. Ghanem, and Mohamed M. Ismail (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19583.txt.
Introduction
Making a trained model forget selected data is a growing need under privacy rules, and a 2024 paper by Omar M. Safa and co-authors compares ways to do it. The team reviewed six leading unlearning methods and applied them to image and text classification models.
Each method was judged on how well it removes the target data, how fast it runs, and how well it meets legal demands. The comparison sets out the strengths and limits of each route in real settings, where a full retrain is often too costly. The authors aim to guide teams that must honor deletion requests without wrecking model quality, and to point out where the field still falls short.
Catalog record
| CREATOR | Omar M. Safa, Mahmoud M. Abdelaziz, Mustafa Eltawy, Mohamed Mamdouh, Moamen Gharib, Salaheldin Eltenihy, Nagia M. Ghanem, and Mohamed M. Ismail |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19583
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Comparative Study of Machine Unlearning Techniques for Image and Text Classification Models Omar M. Safa, Mahmoud M. Abdelaziz, Mustafa Eltawy, Mohamed Mamdouh, Moamen Gharib, Salaheldin Eltenihy, Nagia M. Ghanem, and Mohamed M. Ismail Abstract Machine Unlearning has emerged as a critical area in artificial intelligence, addressing the need to selectively remove learned data from machine learning models in response to data privacy regulations. This paper provides a comprehensive comparative analysis of six state-of-the-art unlearning techniques applied to image and text classification tasks. We evaluate their performance, efficiency, and compliance with regulatory requirements, highlighting their strengths and limitations in practical scenarios. By systematically analyzing these methods, we aim to provide insights into their applicability, challenges, and trade-offs, fostering advancements in the field of ethical and adaptable machine learning. 1 Introduction The rapid expansion of machine learning (ML) applications has raised significant concerns over data privacy. With the integration of ML models across industries, regulations like the GDPR and CCPA mandate that individuals can request the removal of their personal data. This creates a challenge for organizations to comply with privacy laws while maintaining model performance. Traditional ML training often results in models memorizing sensitive data, and removing this data typically requires retraining from scratch, which is resource-intensive. Machine unlearning has been proposed as a solution to allow models to “forget” specific data efficiently without complete retraining, preserving performance and ensuring privacy compliance. Previous work on unlearning in classification models has categorized the problem into three distinct scenarios: (1) Full-class unlearning , which entails removing an entire class from the model, with the forget set containing all instances of a specific class; (2) Sub-class unlearning , which focuses on forgetting a subset of instances within a single class; and (3) Random forgetting , which involves removing arbitrary instances across multiple classes. Despite progress in this field, a critical gap remains in the literature: the absence of a comprehensive, unified study comparing unlearning techniques across diverse datasets and models. Existing research typically evaluates methods on specific datasets or tasks, leading to fragmented insights and a lack of standardized benchmarks for performance comparison. This paper aims to bridge this gap by conducting a comparative analysis of prominent machine unlearning techniques in the context of image and text classification tasks. By evaluating these methods across multiple datasets and offering a unified framework for comparison, we aim to provide valuable insights into the trade-offs, strengths, and weaknesses of each approach. 2 Related Work Foster, Schopf, and Brintrup [ 5 ] introduced an unlearning algorithm called Selective Synaptic Dampening SSD that uses weight sensitivity to adjust model parameters selectively for forgetting specific data. The method identifies weights heavily influenced by the ”forget set” compared to the remaining data and dampens these weights, reducing their impact on predictions. Weight sensitivities are computed using the Fisher Information Matrix (FIM), enabling targeted adjustments to the most sensitive model parameters. Graves, Nagisetty, and Ganesh [ 6 ] proposed an unlearning method called Mislabel Unlearning , a simple approach in which the labels of data in the forget set are randomly changed to those of other classes. The modified data is then used to fine-tune the model for a few epochs, enabling selective forgetting. Chundawat, et al [ 2 ] proposed an unlearning method called Incompetent Teacher that uses selective knowledge transfer between student and teacher models. The student model learns from the forget set under the influence of an incompetent teacher, while a competent teacher provides corrective guidance for the retain set. Kurmanji et al. [ 9 ] proposed SCRUB , an unlearning method based on the teacher-student framework. Similar to the ”Incompetent Teacher” approach, SCRUB uses a student model that learns from a teacher model. However, SCRUB modifies loss function to increase error on the forget set while optimizing accuracy on the retained data. This is achieved through alternating ”max-steps” (focused on forgetting) and ”min-steps” (focused on retaining), with additional steps to restore performance on the retain set. Tarun et al. [ 12 ] introduced the UNSIR algorithm. This approach consists of three steps: generating an error-maximizing noise matrix, impairing the model by training it with this noise on a subset of data, and repairing the model on the retained data. The noise matrix, which maximizes error for the targeted class, is key to disrupting the model’s ability to recall the forgotten data. Following the impair step, the repair step ensures that the model retains its performance on the remaining data. 3 Methodology This study evaluates machine unlearning techniques using image and text classification models, focusing on the trade-off between retaining accuracy on non-forgotten data and effectively forgetting target data. The experiments involve fine-tuning pre-trained models, applying unlearning processes, and measuring performance using key metrics. 3.1 Models and Datasets Image Classification Models: • ResNet18 [ 7 ] : A lightweight residual convolutional network, known for its efficient training. • ViT(google/vit-base-patch16-224) [ 13 ] : Vision Transformer [ 3 ] , which treats image patches as tokens and utilizes transformer architecture. Text Classification Model: • MARBERT [ 1 ] : An Arabic variant of BERT, pre-trained on 128 GB of Arabic text data. Image Datasets: • CIFAR-10 [ 8 ] : Contains 60,000 color images of size 32x32 pixels across 10 classes (e.g., airplanes, cars, birds, etc.). It is used for random forgetting experiments, with 50,000 training and 10,000 test images. • CIFAR-100: A more challenging dataset with 100 classes and 600 images per class. Used for full-class unlearning experiments, it is split into 50,000 training and 10,000 test images. Text Dataset: • Hotel Arabic Reviews Dataset (HARD) [ 4 ] : Contains 93,700 Arabic hotel reviews (only 50,000 were used), labeled from 1 to 5, split into 40,000 training samples and the 10,000 for testing. In our experiments, the models were fine-tuned on the datasets mentioned above. The experiment aimed to perform Full-Class Unlearning and/or Random Forgetting using six unlearning methods: SSD, Incompetent Teacher, SCRUB, UNSIR, and Mislabel. Retraining from scratch was also included as a baseline for comparison. 3.2 Evaluation Metrics The effectiveness of machine unlearning is assessed using the following metrics: 3.2.1 Relative Retain Accuracy ( A_{r} ) Retain accuracy measures the performance of the unlearned model on data not targeted for forgetting. It is computed as: A_{r}=\frac{A_{u}}{A_{b}}\times 100,\ (1) where A_{u} is the accuracy of the unlearned model on the retain set, and A_{b} is the baseline accuracy of the original model. An ideal unlearning method should preserve A_{r} close to 100%. 3.2.2 Relative Forget Accuracy ( A_{f} ) Forget accuracy quantifies the model’s accuracy on the forget set after unlearning. Lower A_{f} values indicate effective forgetting. It is defined as: A_{f}=\frac{A_{u}}{A_{b}}\times 100,\ (2) 3.2.3 Membership Inference Attack (MIA) MIA [ 11 ] evaluates the model’s susceptibility to leaking information about the forget set. Based on logistic regression, MIA returns a probability score indicating the likelihood of a data sample being part of the training set. The optimal MIA value is defined as: An optimal value for MIA might appear to be closer to 0, however as argued by Chundawat et. al [ 2 ] , an abnormally small probability provides as much information to an attacker as an exceedingly large probability. Thus, optimal values for