NOVO: Unlearning-Compliant Vision Transformers

Soumya RoyAmazon IndiaBangalore, India [email protected], Soumya BanerjeeIIT KanpurKanpur, India [email protected], Vinay VermaAmazon IndiaBangalore, India [email protected] Soumik DasguptaWalmart LabsBangalore, India [email protected] Deepak GuptaAmazon IndiaBangalore, India de · 2025

Read NOVO: Unlearning-Compliant Vision Transformers by Soumya RoyAmazon IndiaBangalore, India [email protected], Soumya BanerjeeIIT KanpurKanpur, India [email protected], Vinay VermaAmazon IndiaBangalore, India [email protected] Soumik DasguptaWalmart LabsBangalore, India [email protected] Deepak GuptaAmazon IndiaBangalore, India de (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03281.txt.

Introduction

Soumya Roy and co-authors design an image model that can forget on demand, in a 2025 paper. Laws and ethics may require a trained model to drop the data of a person or class it learned from. The usual fix — retraining or fine-tuning with forget and keep sets — is costly and often weakens the model.

Their design, called NOVO, builds forgetting into vision transformers from the start. During training, each small batch is split into a pretend forget set and a keep set, and the model learns to fail on the forget part. Later, a real request to forget a class can be honored at once, with no extra tuning. Built-in forgetting could make privacy requests cheap to grant.

Catalog record

CREATORSoumya RoyAmazon IndiaBangalore, India [email protected], Soumya BanerjeeIIT KanpurKanpur, India [email protected], Vinay VermaAmazon IndiaBangalore, India [email protected] Soumik DasguptaWalmart LabsBangalore, India [email protected] Deepak GuptaAmazon IndiaBangalore, India de
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03281

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NOVO: Unlearning-Compliant Vision Transformers Soumya RoyAmazon IndiaBangalore, India [email protected], Soumya BanerjeeIIT KanpurKanpur, India [email protected], Vinay VermaAmazon IndiaBangalore, India [email protected] Soumik DasguptaWalmart LabsBangalore, India [email protected] Deepak GuptaAmazon IndiaBangalore, India [email protected] Piyush RaiIIT KanpurKanpur, India [email protected] Abstract Machine unlearning (MUL) refers to the problem of making a pre-trained model selectively forget some training instances or class(es) while retaining performance on the remaining dataset. Existing MUL research involves fine-tuning using a forget and/or retain set, making it expensive and/or impractical, and often causing performance degradation in the unlearned model. We introduce NOVO, an unlearning-aware vision transformer-based architecture that can directly perform unlearning for future unlearning requests without any fine-tuning over the requested set. The proposed model is trained by simulating unlearning during the training process itself. It involves randomly separating class(es)/sub-class(es) present in each mini-batch into two disjoint sets: a proxy forget-set and a retain-set, and the model is optimized so that it is unable to predict the forget-set. Forgetting is achieved by withdrawing keys, making unlearning on-the-fly and avoiding performance degradation. The model is trained jointly with learnable keys and original weights, ensuring withholding a key irreversibly erases information, validated by membership inference attack scores. Extensive experiments on various datasets, architectures, and resolutions confirm NOVO’s superiority over both fine-tuning-free and fine-tuning-based methods. 1 Introduction As AI agents become increasingly pervasive in our everyday lives, concerns about privacy, regulatory compliance, and legal requirements are growing, leading to requests for data removal. Stakeholders may also demand the removal of data instances of specific class(es) or categorie(s) from training dataset, eliminating its impact on the models [ 9 ] . The laws like Right to be Forgotten , GDPR [ 2 ] , and CCPA [ 1 ] further mandate such requirements. A naïve solution involves retraining from scratch, excluding those specific instances. While it yields optimal results, it is computationally expensive. With the rise of LLMs and LVMs [ 28 , 11 , 31 ] like ChatGPT [ 3 ] , LLAMA [ 46 ] , and CLAUDE [ 48 ] , repeated retraining is infeasible. Currently, Machine Learning (ML) models like ViT [ 12 ] and SWIN [ 33 ] are trained for optimal performance without considering possible future unlearning requests. Recently, there has been a growing interest in MUL research [ 17 , 42 , 40 , 49 , 19 , 18 , 29 , 36 , 13 , 32 , 26 , 7 , 41 , 39 , 34 , 23 , 38 , 20 ] , with the existing approaches mainly building on top of standard ML approaches, requiring fine-tuning with the instances for which unlearning has been requested. While this process enables unlearning, it often leads to reduced performance over the retained set, necessitating additional fine-tuning with the retain set, raising concerns about data privacy and storage. Moreover, it also incurs a high cost due to repeated fine-tuning. This prompts an important question: Is it possible to develop an unlearning-compliant model that can forget on demand over the requested classes or sub-classes without the need for explicit fine-tuning with data instances corresponding to the forget classes, while preserving the performance on the retain dataset/classes? This paper addresses the aforementioned question by proposing a novel architecture, NOVO (U N learning-C O mpliant V ision Transf O rmers). Here, we focus on sub-class and full-class unlearning [ 29 , 9 , 42 , 50 , 30 , 35 ] , where instead of forgetting individual samples, the model forgets the entire sub-class(es) or class(es). For instance, in applications like face recognition, a face ID may represent a class, making class-level unlearning useful [ 9 ] . Sub-class forgetting is useful when we want to forget entire sub-class(es) within a set of super-classes. The proposed model is simple and requires minor modifications to the standard architectures, which can be applied to any transformer-based architecture, such as ViT [ 12 ] , CAIT [ 45 ] and SWIN [ 33 ] . The model is trained to be unlearning compliant and aware of potential future unlearning requests. It can forget any set of class(es) or sub-class(es) on-the-fly , just by referencing the set of class/sub-class indices that have been requested to be forgotten. NOVO does not require any forget set or retain set samples and is also free from any fine-tuning or expensive post-processing during the unlearning step. NOVO associates a learnable key with each class or sub-class, enabling the model to predict classes only when the corresponding keys are present. The model has two sets of parameters: learnable keys and the original model weights, both of which are trained jointly to form the final model. Therefore, the original weights alone do not have any prediction ability, which helps mitigate potential attacks and privacy concerns. The removal of class keys results in permanently forgetting a set of classes or sub-classes. To achieve this, the model is trained by simulating learning and unlearning during the training process, where classes in each mini-batch are divided into proxy forget and retain sets, represented by two multi-hot binary vectors. During training, the model receives these vectors along with standard inputs, which helps select the corresponding class/sub-class keys. The composite parameters are optimized to minimize the cross-entropy loss over the retain set, while logits for the forget set are forced to follow a uniform distribution to simulate unlearning. By alternating between the retain and forget sets, this approach effectively simulates various unlearning scenarios. However, iterating over all possible combinations of retain and forget classes requires simulating 2^{C} combinations for C classes, making it infeasible within a limited number of epochs. Therefore, high generalization ability is crucial to handle such unseen configurations. We empirically observe that our unlearning-aware approach to training NOVO generalizes to unseen retain/forget set configurations, automatically returning a class near the vicinity of the forget set. Additionally, we observe that injecting the learnable keys into deeper layers of the network helps learn enriched class or sub-class-specific representations and enhances the model’s performance. Furthermore, we introduce random dropping and expansion of the proxy retain and forget set multi-hot binary vectors to prevent any correlations between the learnable keys and the retained set samples, significantly boosting the model’s robustness and overall accuracy. NOVO can also function as a controllable network, using class or subclass keys to control output predictions, making it suitable for scenarios requiring predictions only on a subset of classes or subclasses. Extensive experiments on multiple datasets involving single and multiple-class forgetting scenarios demonstrate that as the forget set grows, the performance of SOTA baselines degrades while our model remains unaffected. Our main contributions can be summarized as follows: • We propose a novel, on-the-fly, unlearning-aware transformer architecture that can forget the desired set of class(es) or sub-class(es) without requiring any expensive post-processing or training samples. • The deeper key injection, random drop and expand, and unlearning-aware training enhance learning and unlearning. NOVO can work as a controllable network and supports predictions for a subset of class(es). • Extensive experiments on CIFAR10/20/100, and TinyImagenet-200 using ViT [ 12 ] , CAIT [ 45 ] , and SWIN [ 33 ] across resolutions (32 \times 32, 224 \times 224, 384 \times 384) show our model’s efficacy over

Read the complete text