Erase to Retain: Low Rank Adaptation Guided Selective Unlearning in Medical Segmentation Networks

Nirjhor Datta, Md. Golam Rabiul Alam · 2025

Read Erase to Retain: Low Rank Adaptation Guided Selective Unlearning in Medical Segmentation Networks by Nirjhor Datta, Md. Golam Rabiul Alam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16574.txt.

Introduction

Nirjhor Datta and Md. Golam Rabiul Alam present Erase to Retain, a method for making medical image segmentation networks forget selected knowledge on demand, for privacy compliance or dataset revision, without retraining from scratch. The framework teaches a student network to drop lesion- or class-specific representations in low-rank adapter spaces inside the decoder.

Training has two phases. In the strong phase, the adapters are pushed to contradict the teacher's confident predictions on the forget set, wiping the targeted semantics. A gentle phase then restores performance on kept data by refining only the head. On skin lesion data the forget-set overlap score fell sharply while retained and validation scores held or improved, and classification tests showed the same pattern across domains.

Catalog record

CREATORNirjhor Datta, Md. Golam Rabiul Alam
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-16574

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Erase to Retain: Low Rank Adaptation Guided Selective Unlearning in Medical Segmentation Networks Nirjhor Datta, Md. Golam Rabiul Alam Abstract The ability to selectively remove knowledge from medical segmentation networks is increasingly critical for privacy compliance, ethical deployment, and continual dataset revision. We introduce Erase to Retain , a controllable unlearning framework for medical image segmentation that achieves targeted forgetting without full retraining. Our method leverages a teacher-student distillation paradigm with Low-Rank Adaptation (LoRA) constrained subspace updates, enabling the student network to erase lesion or class-specific representations in low-rank decoder spaces while preserving global anatomical understanding. During the strong unlearning phase, LoRA modules are adversarially optimized to contradict the teacher’s confident predictions on a designated forget subset, enforcing semantic removal, followed by a gentle restoration phase that recovers generalization on retained data via head only supervised refinement. For ISIC segmentation, the student suppresses forget-set IoU from 0.875 to 0.509 while maintaining competitive performance on the retain and validation splits ( 0.647–0.677 IoU). On the cross domain CHASE dataset, Erase to Retain consistently lowers forget set IoU while preserving utility on retain and validation sets. On ISIC classification, our method reduces accuracy on the forget subset from 87.0% to 64.1% while improving retain accuracy from 83.9% to 90.6% . These results demonstrate that LoRA-based subspace unlearning provides a practical pathway toward responsible, controllable, and reversible unlearning in medical image analysis allowing models to forget sensitive samples or structures while retaining performance where it matters. The code is available at: Erase to Retain 1 Introduction Deep learning systems deployed in high stakes domains such as healthcare, finance, and security increasingly face the requirement to forget previously learned information upon request. The rise of data privacy regulations, including the right to be forgotten under GDPR and similar emerging global frameworks, mandates that machine learning models provide mechanisms for retrospective removal of specific training samples or cohorts. While significant progress has been made in model unlearning for image classification Ginart et al. 2019 ; Bourtoule et al. 2021 , language models Jagielski et al. 2022 , and recommendation systems Bourtoule et al. 2021 , unlearning for dense prediction tasks remains largely unexplored. Medical image segmentation presents a uniquely challenging setting for unlearning. Unlike classification, where forgetting corresponds to suppressing class level information, segmentation models encode spatially fine grained lesion characteristics across thousands of pixels. These signals are deeply entangled with the network’s hierarchical representation, making naive retraining or noise based suppression both computationally expensive and often ineffective Yuan et al. 2019 . Moreover, lesion segmentation carries profound clinical implications: a model that has not reliably forgotten patient specific patterns risks leaking private information through reconstructions Ziller et al. 2020 , attribution maps, or memorized structures. Existing unlearning methods can be broadly categorized into (1) sample level gradient inversion Graves et al. 2021 ; Ullah et al. 2021 , (2) approximate retraining via influence based adjustment Koh and Liang 2017 , and (3) knowledge decomposition or masking Bhaila et al. 2025 . However, these methods either assume classification outputs, require full model retraining, or fail to prevent partial feature retention that still exposes private information. Furthermore, they do not address the unique structure of medical segmentation models, where forgetting must remove patient specific lesion morphology without degrading general lesion detection capabilities essential for clinical reliability. To address these limitations, we introduce Erase to Retain , a principled and lightweight unlearning framework for medical image segmentation. Instead of modifying the entire model, we decompose the network into a frozen backbone and a set of trainable low rank LoRA adapters Hu et al. 2022 . Forgetting is achieved by optimizing only the LoRA parameters using a two phase objective: (i) a retain preserving loss combining supervised segmentation, knowledge distillation, and teacher–student guarding; and (ii) a forget promoting objective that explicitly suppresses lesion activation on the forget set through background only supervision. This design ensures that the base model’s general knowledge remains intact while forgotten lesions are irreversibly removed from the adapted model. Unlike prior work, our method is computationally efficient requiring less than 3\% of trainable parameters while providing a precise mechanism to control the trade off between retention and forgetting. We perform extensive experiments on the ISIC 2018 skin lesion benchmark and demonstrate that Erase to Retain framework (1) eliminates lesion segmentation capability on the forget subset, (2) maintains high accuracy on retained patients, and (3) preserves generalization on held out validation images. Our analysis further reveals that forgetting manifests as a sharp reduction in lesion–pixel sensitivity while the structural fidelity of healthy skin regions remains unchanged. Our contributions are summarized as follows: • We present the first LoRA based unlearning framework for dense medical segmentation, enabling efficient and controllable removal of patient specific lesion information. • We introduce a dual objective loss integrating supervised segmentation, knowledge distillation, and backgrounddriven forgetting, which produces clean and clinically interpretable forgetting behavior. • We provide comprehensive evaluations including pixel level metrics, activation visualizations, retention forgetting tradeoff curves, and classwise analyses revealing that Erase to Retain achieves strong forgetting with minimal retention degradation. • We show that the proposed framework naturally extends to image classification, highlighting its ability to generalize unlearning beyond segmentation. 2 Related Work 2.1 Machine Unlearning Machine unlearning seeks to remove the influence of specific data from trained models without full retraining. Early approaches relied on exact data deletion through sharded training and summation of per-sample updates Cao and Yang 2015 . Subsequent methods focused on certifiable guarantees, including influence function based removal Guo et al. 2019 , Newton-based approximate unlearning Golatkar et al. 2020 , and exact removal through statistical query reformulation Ginart et al. 2019 . Recent works have emphasized practical neural unlearning pipelines, such as gradient ascent forgetting Graves et al. 2021 , retraining efficient scrubbing Raghavan and Balaprakash 2021 , and distillation based forgetting Wang et al. 2024 . However, nearly all prior research has been conducted in NLP or image classification. None of these methods address the pixel level sensitivity and spatial consistency required for medical segmentation. Our work fills this gap by introducing a LoRA based unlearning operator that preserves structural priors while erasing lesion-specific representations. 2.2 Unlearning in Large Language Models Large language models have motivated a new wave of selective forgetting mechanisms. Measuring memorization and private data leakage Carlini et al. 2021 led to follow-up works on erasing sensitive information using model editing Sharma et al. 2024 ; Meng et al. 2022 , knowledge boundary shifting Meng et al. 2022 , and harmful fact removal Liu et al. 2025 . While effective for textual knowledge, these methods assume discrete tokenized representations and do not translate to spatially dense medical images. Our design instead leverages low-rank adaptation layers to modulate feature subspaces associated with the forget set. 2.3 LoRA and Parameter-Efficient Adaptation Low-Rank Adaptation (LoRA) Hu et al. 2022 has emerged as a dominant strategy for parameter-efficient fine-tuning of large models, enabling rapid learning while freezing the base backbone. Extensions such as AdaLoRA Zhang et al. 2023 and DyLoRA Valipour et al. 2023 adapt rank dynamically or conditionally. While LoRA is widely used for efficient adaptation in vision transformers Jia et al. 2022 , diffusion models Ruiz et al. 2023

Read the complete text