Reminiscence Attack on Residuals: Exploiting Approximate Machine Unlearning for Privacy Yaxin Xiao 20034165r@connect.polyu.hk, Qingqing Ye qqing.ye@polyu.edu.hk, Huadi Zheng zhenghuadi@huawei.com, Haibo Hu haibo.hu@polyu.edu.hk, Zi Liang zi1415926.liang@connect.polyu.hk, Haoyang Li hao-yang9905.li@connect.polyu.hk, Yijie Jiao 22103037d@connect.polyu.hk, Department of Electronical and Electronic Engineering, The Hong Kong Polytechnic University Huawei Technologies Co., Ltd., Research Centre for Privacy and Security Technologies in Future Smart Systems Corresponding author: haibo.hu@polyu.edu.hk Abstract Machine unlearning enables the removal of specific data from ML models to uphold the right to be forgotten . While approximate unlearning algorithms offer efficient alternatives to full retraining, this work reveals that they fail to adequately protect the privacy of unlearned data. In particular, these algorithms introduce implicit residuals which facilitate privacy attacks targeting at unlearned data. We observe that these residuals persist regardless of model architectures, parameters, and unlearning algorithms, exposing a new attack surface beyond conventional output-based leakage. Based on this insight, we propose the Reminiscence Attack (ReA) , which amplifies the correlation between residuals and membership privacy through targeted fine-tuning processes. ReA achieves up to 1.90 × 1.90\times and 1.12 × 1.12\times higher accuracy than prior attacks when inferring class-wise and sample-wise membership, respectively. To mitigate such residual-induced privacy risk, we develop a dual-phase approximate unlearning framework that first eliminates deep-layer unlearned data traces and then enforces convergence stability to prevent models from “pseudo-convergence”, where their outputs are similar to retrained models but still preserve unlearned residuals. Our framework works for both classification and generation tasks. Experimental evaluations confirm that our approach maintains high unlearning efficacy, while reducing the adaptive privacy attack accuracy to nearly random guess, at the computational cost of 2 − 12 % 2-12\% of full retraining from scratch. 1 Introduction As the significance of individual privacy grows, numerous laws and regulations have been established to grant individuals the right to protect their privacy. Notably, the General Data Protection Regulation (GDPR) [ 32 ] and the Personal Information Protection and Electronic Documents Act (PIPEDA) [ 1 ] incorporate the right to be forgotten , requiring organizations to delete user data upon request. In addition to traditional database systems, machine learning (ML) systems, which are trained on vast amounts of private data, must comply with this right by removing data from their training sets. This is critical because ML systems are vulnerable to data privacy breaches, such as membership inference attacks [ 38 , 45 ] and model inversion attacks [ 28 ] . The techniques that support such data removal in ML systems are collectively called machine unlearning (MU) [ 6 ] . State-of-the-art (SOTA) machine unlearning methods can be categorized into exact unlearning and approximate unlearning. Exact unlearning necessitates retraining the model from scratch using only the remaining dataset, which is resource-intensive even with optimization techniques [ 6 ] . In contrast, approximate machine unlearning (AMU) modifies the parameters of pre-trained machine learning models to approximate the retained model, allowing selective forgetting while maintaining efficiency. Currently, various AMU methods [ 16 , 24 , 44 , 10 , 47 , 23 ] have achieved near-retrained unlearning efficacy when measured by legacy metrics such as prediction accuracy [ 24 ] . Due to their significantly lower resource demands, AMU algorithms have become a focal point in this field. However, while machine unlearning initially aims to protect the privacy of the target data, it may inadvertently introduce new privacy risks. Such risks come from the imprint of unlearned data left behind in unlearned models, which becomes a source of privacy leakage. According to the imprint types, existing privacy attacks against unlearned data can be divided into two categories: version-dependent and version-independent attacks. Version-dependent privacy attacks exploit updates between historical model versions to infer the privacy of unlearned data [ 9 , 21 , 5 ] . Thus, they are applicable only in scenarios where historical model versions are released. The second type, version-independent attacks, applies to broader scenarios and hence serves as the focus of this study. These attacks [ 27 ] detect privacy leaks by analyzing differences in the model’s behavior on unlearned data versus non-training data. Yet, we argue that current version-independent methods [ 7 , 27 ] only partially exploit the problem. This paper investigates a critical oversight in version-independent privacy leakage analysis: Do existing approximate machine unlearning (AMU) methods inadvertently leak unlearned data privacy through implicit residuals beyond the unlearned model’s output distributions ? Prior works [ 7 , 27 ] focus on output-space residuals yet fail to capture the full extent of privacy risks. We reveal that AMU methods overlook latent residuals that alter their loss landscapes, leading to even greater risks for the membership privacy of unlearned data. To exploit these residuals, we introduce the Reminiscence Attack (ReA), which leverages a targeted fine-tuning approach to detect whether specific classes or samples were part of the training set. To address such privacy leakages, we propose Orthogonal Unlearning & Replay (OUR), an AMU method designed to scrub latent residuals in the model. OUR is built on two key insights: First, class-wise residuals persist because the unlearned model retains high intra-class correlation in hidden representations for the unlearned class, even when its predictions resemble those of retrained models. For deeper scrubbing, we enforce orthogonality between the hidden representations of unlearned data and their original values. Second, sample-wise residuals arise because the unlearned model stays in a “pseudo-converged” states that mimic the retrained model outputs (e.g., the prediction accuracy) but preserves latent information of unlearned data. To counteract this, we separate OUR into two phases to ensure model convergence through a final fine-tuning phase. In summary, our contributions are as follows: • We introduce Reminiscence Attack (ReA), a novel version-independent attack which targets the membership privacy of unlearned samples and classes. ReA uncovers privacy leakage beyond what existing attacks [ 27 , 24 ] detect by exploiting residuals in the model’s loss landscape. • To address the privacy vulnerabilities exposed by ReA, we propose an approximate unlearning algorithm, Over-Unlearning & Replay (OUR), designed to eliminate latent residuals within the unlearned models. • Extensive experiments evaluate the unlearning efficacy of approximate unlearning algorithms and their privacy leakage using our ReA attacks, with results in classification scenarios summarized in Figure 1 . ReA effectively exposes unlearned data privacy, achieving up to 90\% attack accuracy for unlearned class membership inference. The results also confirm that OUR significantly reduces ReA’s attack accuracy to near-random levels while preserving high unlearning efficacy and efficiency. (a) (b) Figure 1 : Comparison of unlearning efficacy (Tug-of-War metric, Equation 1 ) and ReA attack accuracy across 12 unlearning methods tested in Section 5.2 on CIFAR20 (11 baselines and our OUR). OUR method approaches the “Retrain” in the down-right corner, showing high unlearning efficacy and low privacy leakage. 2 Problem Definition 2.1 Machine Unlearning (MU) Machine unlearning (MU) removes the influence of specific training data on a pre-trained machine learning model F(\cdot;\theta) through exact or approximate methods. While exact unlearning requires retraining from scratch, approximate unlearning modifies model parameters \theta to achieve comparable unlearning efficacy at significantly lower computational cost. Formally, given a model F(\cdot;\theta) trained on dataset \mathcal{D} , let \mathcal{D}_{u} denote unlearned data and \mathcal{D}_{r}=\mathcal{D}\setminus\mathcal{D}_{u} to the retained data. The approximate unlearning algorithm \mathcal{U} aims to produce parameters \theta^{u}=\mathcal{U}(\theta) such that F(\cdot;\theta^{u}) approximates the model F(\cdot;\theta^{0}) retrained on \mathcal{D}_{r} . Unlearning in Classification . In classification tasks, unlearning operates at two granularities: sample-wise unlearning for individual data points and class-wise unlearning for entire classes [ 16 ] . Specifically, sample-wise unlearning processes \mathcal{D}_{u} as discrete samples, requiring the unlearned model F(\cdot;\theta^{u}) to match the retrained model F(\cdot;\theta^{0}) ’s behavior by treating \mathcal{D}_{u} as unseen test data. Class-wise unlearning handles \mathcal{D}_{u} as entire classes, and demands F(\cdot;\theta^{u}) aligns with F(\cdot;\theta^{0}) to recognize unlearned classes as out-of-distribution (OOD) classes. This ensures the unlearned model has identical task scopes to F(\cdot;\theta^{0}) . Unlearning in Image Generation . For generative models such as text-to-image diffusion models [ 35 ] , we shift focus from safety alignment to privacy preservation. While existing methods focus on removing harmful contents (e.g., nudity) [ 13 , 14 ] , we address identity-specific concept unlearning in portrait generation. This process aims to eliminate identity-specific concepts from F(\cdot,\theta) while maintaining output quality for retained tasks. Legacy MU Metrics . Machine unlearning is assessed with the following legacy metrics [ 24 , 44 , 29 ] : 1 Test Accuracy (TA) . It is defined as TA =Acc_{\mathcal{D}_{t}}(F(.;\theta^{u})) , which measures the accuracy of F(.;\theta^{u}) on the test dataset \mathcal{D}_{t} . 2 Unlearning Accuracy (UA) . It is calculated by UA =Acc_{\mathcal{D}_{u}}(F(.;\theta^{u})) . 3 Retaining Accuracy (RA) . It’s defined as RA =Acc_{\mathcal{D}_{r}}(F(.;\theta^{u})) . The closer the three accuracy metrics are to the ‘ground-truth’ retrained model F(.;\theta^{0}) , the better the unlearning efficacy. 4 Membership Inference Attack on \mathcal{D}_{u} (MIA Efficacy) . This metric quantifies the success rate of membership inference attacks [ 38 ] in detecting \mathcal{D}_{u} as training data for F(\cdot;\theta^{u}) . Unlike prior evaluations [ 24 ] measuring only the true positive rate (TPR), we adopt TPR at a fixed 0.1 false positive rate (FPR) for stricter privacy assessment. 5 Run-Time Efficacy (RTE) . This metric refers to the time a machine unlearning algorithm consumes, which assesses its practical efficiency. 6 Tug-of-War (ToW) [ 47 ] . This unified metric quantifies the alignment between unlearned ( \theta^{u} ) and fully retrained ( \theta^{0} ) models across multiple metrics M_{1} , M_{2} , and M_{3} . When instantiated with TA, UA, and RA, ToW measures unlearning efficacy. Let d(\cdot,\cdot) denote the absolute distance between metric values. The ToW score is defined as: \footnotesize\text{ToW}(M_{1},M_{2},M_{3})=\prod\nolimits_{i=1}^{3}(1-\frac{d(M_{i}(\theta^{u}),M_{i}(\theta^{0}))}{M_{i}(\theta^{0})}).\vskip-1.9919pt (1) 2.2 Privacy Attacks Targeting MU Privacy attacks compromise unlearned data privacy via exploring information leakage in machine unlearning processes. These attacks fall into two categories: (1) version-dependent attacks, which analyze model update histories [ 9 , 21 , 5 ] , and (2) version-independent attacks, which leverage residuals of approximate unlearning. As version management addresses only the former, we focus on the latter category which exposes vulnerabilities in approximate unlearning methods. This category includes two benchmarks: (1) MIA-LiRA . It applies SOTA membership inference attacks (MIA) LiRA [ 7 , 3 ] , which uses shadow models and a likelihood ratio test on model confidence to detect unlearned data. Since its attack model is designed to detect training data for the original model F(\cdot;\theta) , its effectiveness diminishes when unlearned and training data yield distinct output distributions. (2) MIA against User Privacy (MIA-UP) [ 27 ] . This attack distinguishes unlearned data from test data. While adopting LiRA’s shadow model framework, this method uniquely synchronizes shadow models with the unlearned model’s unlearning process. Then, the binary classifier attacker trains on the outputs of shadow models, using unlearned data as positives and test data as negatives. However, it becomes unstable if shadow and unlearned models’ unlearning processes diverge. Up to now, privacy attacks targeting unlearned data primarily exploit residuals in output distributions. 2.3 Threat Model While existing privacy attacks on machine unlearning focus on sample-level membership inference, we pioneer the formalization of class membership privacy leaks, which infer whether specific classes were ever unlearned. We assume adversaries are aware of the task scope and have either white-box or black-box API access. Sample-wise Membership Inference Attack (MIA) . For models undergoing sample-wise unlearning ( F(\textperiodcentered;\theta^{u} )), adversaries infer membership of individual unlearned samples, where test data is negative references. Following SOTA MIA evaluations [ 7 , 4 ] , we analyze the true-positive rate (TPR) versus the false-positive rate (FPR) of attacks (i.e., the precision-recall tradeoff) rather than fixed accuracy. Specifically, the attack model outputs continous confidence scores through \mathcal{A}^{\prime} , which are then thresholded by \tau to yeild a membership prediction. Formally, for an inferred sample x with label y , the output of the sample-MIA attacker \mathcal{A}_{\text{sample}} is: \footnotesize\mathcal{A}_{\text{sample}}(x,y)=\mathbf{1}[\mathcal{A}^{\prime}(F(x;\theta^{u}),y)>\tau],\vskip-2.84544pt (2) where \mathbf{1} is the indicator function and \tau controls the precision-recall tradeoff. Class-wise Membership Inference Attack (MIA) . For models undergoing F(.;\theta^{u}) undergoes class-wise unlearning, the adversary infers membership of candidate OOD classes \mathcal{D}^{\text{ood}} , i.e., determining whether it was unlearned. Crucially, this attack can extend to concept unlearning in generative tasks, targeting the membership of forgotten concepts. Mirroring the sample-wise formulation (Equation 2 ), the output of the class-wise MIA attacker \mathcal{A}_{\text{class}} for \mathcal{D}^{\text{ood}} is: \footnotesize\mathcal{A}_{\text{class}}(\mathcal{D}^{\text{ood}})=\mathbf{1}[\mathcal{A}^{\prime}(\mathcal{D}^{\text{ood}})>\tau].\vskip-5.69046pt (3) 3 Reminisence Attack (ReA): Exploiting Loss Landscape Residuals We propose the Reminiscence Attack (ReA), a novel version-independent membership inference attack against unlearned samples, classes, and concepts. This method exploits latent residuals preserved in the loss landscape during approximate unlearning, establishing an orthogonal attack surface to existing approaches (MIA-LiRA/UP). Understanding Unlearning Residuals in Loss Landscapes . Approximate unlearning may produce measurable residuals that deviate from ‘gold-standard’ retrained models, causing persistent privacy risks. Formally, for an unlearned model F(\cdot;\theta^{u}) obtained by unlearning dataset \mathcal{D}_{u} , we define its residual as \footnotesize R(\mathcal{D}_{u}):=\mathbb{E}\left[d(F(\mathcal{D}_{u};\theta^{u}),F(\mathcal{D}_{\text{non-tr}};\theta^{u}))\right], (4) where \mathcal{D}_{\text{non-tr}} is the non-training counterpart to \mathcal{D}_{u} . d is a task-specific metric measuring model behavior discrepancy. We reveal that these residuals not only manifest in output distributions but also reshape the loss landscape. As visualized in Figure 2(a) , unlearned models remain near optima (bright regions) for \mathcal{D}_{u} data and converge rapidly under guided finetuning, revealing latent traces, contrasting with non-training data (including OOD classes and test samples). To efficiently expose membership information in the loss landscape, targeted fine-tuning is the natural choice, which we term reminiscence . Its implementation depends on the inference target. In class-wise unlearning, as shown in Figure 2(a) (left), the unlearned model lies on flat plateaus near sharp minima. Guiding fine-tuning toward the minima accelerates convergence for unlearned classes while slowing non-training classes, which reveals membership. This guidance is achieved by introducing retained data in fine-tuning, as the targeted path aligns with their optimal solutions. In sample-wise unlearning, as shown in Figure 2(b) (left), the unlearned model remains on a steep loss landscape for unlearned data, indicating incomplete convergence. If the model shifts to the low-loss basin (bright regions), the loss gap between unlearned and test data widens, further exposing membership. Unlike class-wise ReA, inferred samples do not share memberships and cannot be fine-tuned in groups. However, fine-tuning on retained samples still guides the model to the low-loss basin, as their optimal solution regions align with unlearned samples. While both scenarios rely on retained samples, Section 3.1 explains how to remove this assumption. (a) (b) Figure 2 : Visualized loss landscapes around unlearned models in CIFAR20 experiments. Dashed lines are guided fine-tuning trajectories. Appendix C provides additional results on other datasets. 3.1 Reminiscence Attack (ReA) Earlier, we introduced the method of reminiscence (targeted fine-tuning) to leverage residuals in the loss landscape to amplify the membership leakage of unlearned data. Here, we detail the steps of the reminiscence attack (ReA). Class-wise ReA . To detect the link between residuals and an unlearned class, ReA reintroduces the inferred class (a set \mathcal{D}^{\text{ood}} ) into training and measures how quickly it resonates with residuals to derive the attack confidence score. Formally, this is quantified by the resonance index Idx r , defined as the number of iterations required for the model to achieve high prediction accuracy (i.e., convergence threshold) on \mathcal{D}^{\text{ood}} . Then, the confidence score in Equation 3 is computed as: \footnotesize\mathcal{A}^{\prime}(\mathcal{D}^{\text{ood}})=1-\text{Idx}_{\text{r}}(\mathcal{D}^{\text{ood}})/\text{Idx}_{\text{max}},\vskip-2.84544pt (5) where \text{Idx}_{\text{max}} is the total number of training iterations. A lower Idx r means faster convergence and suggests the inferred class closer to the unlearned class in Figure 2(a) , which leads to a higher confidence score in Equation 5 when inferred as unlearned data. The above method assumes a white-box attacker, but this assumption can be relaxed with substitute models from model extraction [ 40 ] , given the demonstrated transferability of residuals in Appendix E.8 . There are two key challenges affecting class-wise ReA’s efficiency. First, ReA ideally requires the retained dataset to guide the shortest optimal path, but attackers may lack access to it. This issue can be mitigated by minimizing logits updates from OOD samples (excluding the inferred class) [ 30 ] , as this fulfills the retained dataset’s key role in maintaining utility. Second, the fine-tuned learning rate (lr) significantly impacts ReA’s stability. To address this, we introduce the multi-lr aggregation strategy , which averages resonance indexes across multiple learning rates \{\text{lr}_{j}\}_{j=1}^{J} . Thus, the revised confidence score is: \footnotesize\mathcal{A}^{\prime}(\mathcal{D}^{\text{ood}})=1-\left[\sum\nolimits_{j=0}^{J}{\text{Idx}_{\text{r}}(\mathcal{D}^{\text{ood}},\text{lr}_{j}})/(J\cdot\text{Idx}_{\text{max}})\right].\vskip-1.9919pt (6) Sample-wise ReA . In sample-wise unlearning, residuals persist primarily due to incomplete convergence. To fully expose the membership of unlearned samples, ReA fine-tunes the unlearned model on the retained dataset with a reduced learning rate for convergence. If residuals remain, the model’s accuracy on unlearned samples rises abnormally higher than on test data, signaling additional privacy leakage. Building on this insight, ReA follows two steps: (1) the reminiscence process , where the unlearned model is fine-tuned on the retained dataset, and (2) the MIA process , where membership is inferred using MIA-LiRA [ 7 ] . MIA-LiRA is preferred over MIA-UP to avoid dependence on unlearning algorithm knowledge. However, the first step requires knowledge of the retained data identity, which is often inaccessible to attackers. To address this, we propose inferring a set of high-confidence positive samples using MIA-LiRA, and them as a “pseudo” retained dataset in the first step. If white-box access is unavailable, sample-wise ReA reduces to MIA-LiRA. Appendix A provides pseudo codes for class-wise and sample-wise ReA. 4 Scrubbing Residuals in Unlearning: Orthogonal Unlearning & Replay (OUR) Previous analysis indicates that if residuals remain from approximate machine unlearning (AMU) processes, particularly in the overlooked loss landscape, they may unintentionally facilitate membership inference attacks (e.g., ReA) against unlearned samples. This highlights the necessity to develop AMU methods that not only align output distributions with retrained models but also scrub residuals, to ensure privacy protection for unlearned data. 4.1 High-level Solution of Scrubbing Residuals To scrub residuals, we must understand their cause and address both class-wise and sample-wise cases. Class-wise residuals persist when the collective features of the unlearned class remain in deeper hidden layers (far from the output layer), as confirmed in Section 5.3 . To eliminate these traces, we enforce orthogonality between the hidden and original features of unlearned samples, achieving deeper scrubbing beyond existing methods (e.g., random label [ 17 ] ). This process is termed as orthogonal unlearning. For sample-wise cases, residuals arise from incomplete convergence on retain tasks, where unlearned accuracy (UA) falsely resembles test samples’ behavior. Resolving this requires the model to reach an actual convergence state. A single loss function optimizing both and model utility often hinders this process, so we structure unlearning in two phases : (1) Orthogonal Unlearning to deeply scrub unlearned sample information, and (2) replay on the retained set to restore convergence and preserve utility. Our notion of orthogonality differs from prior orthogonal projection unlearning [ 43 , 20 , 15 ] , which perturbs directions orthogonal to retaining tasks to preserve utility. Instead, we minimize the cosine similarity between unlearned features and their original state to scrub unlearning residuals. 4.2 Orthogonal Unlearning & Replay (OUR) Next, we detail the two phases of OUR. Let F_{l}(:;\theta) represent the output of the l -th layer of the unlearned model (i.e., representations/feature), and \theta^{0} be the pre-unlearning states. In the first phase, the model is trained over e_{1} epochs with the following loss function: \footnotesize L_{\text{orth}}=\sum\nolimits_{(x,y)\in\mathcal{D}_{u}}\sum\nolimits_{l\in\{l\}_{k}}[F_{l}^{T}(x;\theta)F_{l}(x;\theta^{0})]^{2}.\vskip-2.84544pt (7) L_{\text{orth}} encourages orthogonality between the the features of F(\cdot;\theta) and F(\cdot;\theta^{0}) on unlearned samples. Here, \{l\}_{k} denotes the set of k hidden layers selected for this phase. During the replay phase, the unlearned model is fine-tuned on D_{r} for e_{2} epochs, yielding the final unlearned model F(.;\theta^{u}) . To enhance unlearning effectiveness, we incorporate l1 regularization optimization [ 24 ] . The pseudo-code for OUR is provided in Appendix A . Key Considerations of OUR . How is run-time efficacy (RTE) maintained? RTE is ensured by excluding the retained set in the orthogonal unlearning phase. While this may induce immediate utility degradation, the replay phase achieves rapid recovery due to the minimal change of neurons. Our analysis in Appendix B shows that most neurons undergo trivial changes during unlearning, allowing efficient recovery in the replay phase. How to prevent model collapse during orthogonal unlearning? Model collapse occurs when parameter gradients explode. Therefore, the orthogonal unlearning phase should constrain the maximal parameter change \Delta_{\text{max}}\leq\Delta_{\text{thr}} . \Delta_{\text{thr}} is a value (e.g., 5e-3 ) which may trigger gradient exposure during replay and set as the maximum distance between two random models (see Appendix B ). 5 Experimental Evaluation In this section, we first evaluate the privacy vulnerabilities of 11 approximate unlearning benchmarks using Reminiscence Attack (ReA). We then analyze the unlearning efficacy and privacy leakage of the proposed Orthogonal Unlearning & Replay (OUR) framework. Our code is available at https://github.com/orthogonalunlearning-replay/OUR . 5.1 Setup Datasets and Models . We conduct classification experiments on three benchmarks: CIFAR10 with ResNet18, CIFAR100 with ResNet18 [ 19 ] , and CIFAR20 (20 broad classes aggregated from CIFAR100’s fine-grained labels) with Vision Transformer (ViT) [ 12 ] . Class-wise unlearning is performed on CIFAR100 and CIFAR20, while sample-wise unlearning applies to CIFAR10 and CIFAR20. In the image-generation domain, we specialize in unlearning for text-to-image Stable Diffusion (SD) [ 35 ] with a focus on privacy. Specifically, we perform identity-level unlearning on an SD model fine-tuned via DreamBooth [ 36 ] using CelebA-HQ [ 26 ] identities. Unlearning Configuration . We benchmark 6 SOTA AMU algorithms and 5 optimized frameworks (Appendix D.2 ). For OUR, \{l\}_{k} uses the first, middle, and last block outputs, as verified in Appendix E.1 . Other detailed OUR setups are in Appendix D.1 . For class-wise unlearning, each trial selects five random classes as out-of-distribution (OOD) and one target class for unlearning. Using only one OOD class could introduce bias due to class-specific differences. Thus, corresponding experiments are labeled CIFAR20 (-5) and CIFAR100 (-5). MU evaluation metrics follow Section 2 . Privacy Attack Parameters . Class-wise ReA adversaries have limited access to unlearned data, with only 3% for CIFAR20 and 20% for CIFAR100. Each attack experiment consists of 50 trials with randomized class selections, totaling 300 inferred classes. ReA uses an SGD optimizer and an lr set \{0.001,0.005,0.007,0.01\} to perform a multi-lr aggregration strategy. Its convergence threshold is set to 75\% . The ablation studies discussing hyperparameters of ReA are presented in Appendix E.3 and Appendix E.4 . For sample-wise MIA, following prior work [ 27 ] , we assume adversaries have a dataset containing training (retained), unlearned, and test data but lack knowledge of their memberships. Here, MIA specifically focuses on the membership of unlearned data rather than training (retained) data. Sample-wise ReA uses an lr of 0.01 for CIFAR10 (SGD) and 5 e -5 for CIFAR20 (Adam), which is 0.1\times the training lr. Evaluation for MIA follows SOTA ROC analysis [ 7 ] , measuring MIA performance with TPR at a fixed FPR. Balanced attack accuracy is also reported. Detailed descriptions of OUR hyper-parameters, ReA settings, and platforms are provided in Appendix D.1 . Privacy Attack Benchmarks . To compare with ReA, we evaluate two SOTA attacks: MIA-LiRA [ 7 ] and MIA-UP [ 27 ] , as described in Section 2 . Though originally designed for sample-wise MIA, we adapt them to class-wise MIA as follows. Both methods run on the entire set of inferred classes which match the training set size, producing a single positive rate with a fixed threshold \tau^{\prime} . This score serves as the output of \mathcal{A}^{\prime} in Equation 3 . Specifically, MIA-UP replaces shadow models’ test samples with OOD classes to serve as the negative samples. MIA-LiRA assumes knowledge of model architecture and hyper-parameters, while MIA-UP further requires unlearning algorithm details. 5.2 Evaluation Results in Classification Tasks We present class-wise and sample-wise unlearning performance for classification tasks in Tables 1 , with subscripts indicating standard deviations and blue values showing differences from the retrained model. The corresponding class-wise and sample-wise privacy attack results for different AMU methods are shown in Table 2 and Table 3 , with bold denoting the best results. Table 1 demonstrates that our OUR achieves unlearning efficacy comparable to the ground-truth retrained model on both ViT and ResNet architectures, while exhibiting superior run-time efficiency (RTE) which is lower than 8\% of retraining time. Besides, the privacy vulnerability evaluation reveals two critical insights (Tables 2 - 3 ). First, our Reminiscence Attack (ReA) significantly outperforms existing privacy attacks (MIA-LiRA and MIA-UP) across most MU methods. Class-wise MU methods such as random label (RL) [ 17 ] , and its optimized version Salun [ 13 ] and RUM [ 47 ] , IU [ 24 ] , and BU [ 8 ] exhibit severe vulnerabilities, with ReA achieving over 81\% accuracy. This vulnerability likely arises because RL-based methods mainly affect model parameters near the output layer, and IU and BU have limited unlearning efficacy on ViT models. For sample-wise unlearning, IU [ 24 ] and FF [ 16 ] which lack gradient descent processes, and gradient ascent (GA) [ 39 ] show heightened susceptibility to the reminiscence process, with ReA attaining over 55\% (on CIFAR10) and 70\% (on CIFAR20) accuracy. Second, our OUR method substantially mitigates privacy leakage and reduces class-wise and sample-wise MIA accuracies lower than 54.73\% and 53.80\% , respectively, which are significantly close to retraining baselines. ROC Analysis of Privacy Attacks . ROC analysis [ 7 ] MIA via the trade-off between true positive rate (TPR) and false positive rate (FPR). The area under the curve (AUC) quantifies attack effectiveness, with higher values indicating better separation between member and non-member distributions. Figures 3 compare ROC curves for MU benchmarks, OUR, and retrained models on CIFAR20 in class-wise MIA and sample-wise MIA, respectively. The results indicate that ReA achieves substantially higher AUC values (up to 1.78\times/1.55\times ) than MIA-LiRA/UP, demonstrating superior membership inference capability. Notably, OUR’s curves align closely with retrained models (AUC difference <0.11 ), suggesting better privacy preservation. Complete ROC analyses of all AMU appear in Appendix E.5 . Table 1 : Machine Unlearning (MU) for Classification Class-wise Unlearning MU TA/% UA/% RA/% MIA Efficacy/% RTE (s) CIFAR20 (-5) - ViT Retrain 51.31 \pm 0.11 ( 0.00 ) 0.00 \pm 0.00 ( 0.00 ) 99.99 \pm 0.01 ( 0.00 ) 1.67 \pm 0.10 ( 0.00 ) 1801.26 FT [ 17 ] 49.40 \pm 0.27 ( 1.91 ) 0.74 \pm 0.58 ( 0.74 ) 97.89 \pm 0.04 ( 2.10 ) 1.54 \pm 0.18 ( 0.13 ) 120.99 GA [ 39 ] 49.13 \pm 0.55 ( 2.18 ) 0.00 \pm 0.00 ( 0.00 ) 99.01 \pm 0.02 ( 0.98 ) 92.05 \pm 3.13 ( 90.38 ) 69.46 RL [ 17 ] 50.72 \pm 0.15 ( 0.59 ) 0.00 \pm 0.00 ( 0.00 ) 99.98 \pm 0.01 ( 0.01 ) 1.66 \pm 0.08 ( 0.00 ) 26.04 IU [ 24 ] 48.21 \pm 0.52 ( 3.10 ) 94.95 \pm 4.46 ( 94.95 ) 92.21 \pm 0.21 ( 7.78 ) 20.98 \pm 2.29 ( 19.31 ) 30.32 BU [ 8 ] 50.20 \pm 1.49 ( 1.11 ) 15.35 \pm 1.23 ( 15.35 ) 88.73 \pm 0.96 ( 11.26 ) 15.40 \pm 0.45 ( 13.73 ) 29.92 SCRUB [ 27 ] 51.83 \pm 0.19 ( 0.52 ) 1.64 \pm 0.14 ( 1.64 ) 97.54 \pm 0.62 ( 2.45 ) 5.00 \pm 0.72 ( 3.33 ) 131.58 l1 Sparse [ 24 ] 51.34 \pm 0.18 ( 0.03 ) 0.00 \pm 0.00 ( 0.00 ) 99.93 \pm 0.02 ( 0.06 ) 15.40 \pm 0.62 ( 13.73 ) 142.95 SalUn [ 13 ] 51.05 \pm 0.21 ( 0.28 ) 3.05 \pm 0.15 ( 3.05 ) 99.96 \pm 0.01 ( 0.03 ) 1.67 \pm 0.52 ( 0.00 ) 67.41 SFRon [ 23 ] 48.18 \pm 0.25 ( 3.13 ) 0.00 \pm 0.00 ( 0.00 ) 95.63 \pm 0.52 ( 4.36 ) 87.43 \pm 2.35 ( 85.76 ) 95.69 RUM [ 47 ] 51.79 \pm 0.12 ( 0.46 ) 2.85 \pm 0.28 ( 2.85 ) 99.88 \pm 0.02 ( 0.03 ) 0.00 \pm 0.00 ( 1.67 ) 98.28 OUR 51.98 \pm 0.13 ( 0.67 ) 0.00 \pm 0.00 ( 0.00 ) 99.97 \pm 0.01 ( 0.02 ) 3.67 \pm 0.11 ( 0.00 ) 78.24 CIFAR100 (-5) - ResNet-50 Retrain 62.63 \pm 0.14 ( 0.00 ) 0.00 \pm 0.00 ( 0.00 ) 99.98 \pm 0.00 ( 0.00 ) 0.00 \pm 0.00 ( 0.00 ) 2718.51 FT [ 17 ] 63.90 \pm 0.29 ( 1.27 ) 0.63 \pm 0.31 ( 0.63 ) 99.98 \pm 0.00 ( 0.00 ) 0.00 \pm 0.00 ( 0.00 ) 123.46 GA [ 39 ] 61.02 \pm 0.19 ( 1.61 ) 0.00 \pm 0.00 ( 0.00 ) 99.96 \pm 0.02 ( 0.02 ) 14.15 \pm 2.24 ( 14.15 ) 57.48 RL [ 17 ] 63.84 \pm 0.15 ( 1.21 ) 0.00 \pm 0.08 ( 0.00 ) 99.98 \pm 0.01 ( 0.00 ) 6.33 \pm 0.68 ( 6.33 ) 42.58 IU [ 24 ] 61.09 \pm 0.23 ( 1.54 ) 0.44 \pm 0.05 ( 0.44 ) 99.95 \pm 0.02 ( 0.03 ) 0.00 \pm 0.00 ( 0.00 ) 16.76 BU [ 8 ] 61.15 \pm 0.14 ( 1.48 ) 0.00 \pm 0.00 ( 0.00 ) 89.55 \pm 3.21 ( 10.43 ) 0.00 \pm 0.00 ( 0.00 ) 49.15 SCRUB [ 27 ] 61.02 \pm 0.35 ( 1.61 ) 0.00 \pm 0.00 ( 0.00 ) 99.97 \pm 0.02 ( 0.01 ) 0.00 \pm 0.00 ( 0.00 ) 136.81 l1 Sparse [ 24 ] 62.13 \pm 0.16 ( 0.50 ) 0.00 \pm 0.00 ( 0.00 ) 98.47 \pm 0.14 ( 1.52 ) 1.33 \pm 0.11 ( 1.33 ) 126.67 SalUn [ 13 ] 62.93 \pm 0.18 ( 0.30 ) 0.00 \pm 0.00 ( 0.00 ) 98.90 \pm 0.08 ( 0.10 ) 6.33 \pm 2.43 ( 6.33 ) 38.29 SFRon [ 23 ] 61.35 \pm 0.12 ( 0.28 ) 0.00 \pm 0.00 ( 0.00 ) 95.76 \pm 1.14 ( 0.04 ) 29.33 \pm 5.47 ( 29.33 ) 150.77 RUM [ 47 ] 62.24 \pm 0.25 ( 0.39 ) 0.00 \pm 0.00 ( 0.00 ) 99.07 \pm 0.12 ( 0.91 ) 0.00 \pm 0.00 ( 0.00 ) 136.44 OUR 62.88 \pm 0.13 ( 0.25 ) 0.00 \pm 0.00 ( 0.00 ) 99.96 \pm 0.02 ( 0.02 ) 0.00 \pm 0.00 ( 0.00 ) 72.12 10% Random Sample Unlearning (Sample-wise Unlearning) MU TA/% UA/% RA/% MIA Efficacy/% RTE (s) CIFAR10 - ResNet-18 Retrain 88.98 \pm 0.39 ( 0.00 ) 88.91 \pm 0.51 ( 0.00 ) 99.75 \pm 0.21 ( 0.00 ) 6.40 \pm 0.25 ( 0.00 ) 2401.89 FT [ 17 ] 87.76 \pm 0.26 ( 1.12 ) 88.36 \pm 0.51 ( 0.55 ) 97.39 \pm 0.25 ( 2.36 ) 6.80 \pm 0.31 ( 0.40 ) 145.52 GA [ 39 ] 89.79 \pm 0.26 ( 0.81 ) 85.73 \pm 2.15 ( 0.55 ) 98.09 \pm 0.06 ( 1.66 ) 11.00 \pm 0.37 ( 4.60 ) 116.41 RL [ 17 ] 88.78 \pm 0.28 ( 0.10 ) 89.39 \pm 0.17 ( 0.48 ) 96.38 \pm 0.13 ( 4.37 ) 8.00 \pm 0.42 ( 1.60 ) 43.65 IU [ 24 ] 85.70 \pm 0.21 ( 3.28 ) 90.02 \pm 0.43 ( 1.11 ) 91.11 \pm 0.79 ( 8.64 ) 12.20 \pm 1.50 ( 5.80 ) 30.32 FF [ 16 ] 83.99 \pm 0.19 ( 4.99 ) 90.39 \pm 0.25 ( 1.48 ) 90.80 \pm 0.36 ( 8.85 ) 9.60 \pm 0.14 ( 3.20 ) 2161.07 SCRUB [ 27 ] 89.40 \pm 0.15 ( 0.42 ) 89.73 \pm 0.21 ( 0.83 ) 97.19 \pm 0.04 ( 2.56 ) 8.40 \pm 0.39 ( 2.00 ) 1353.33 l1 Sparse [ 24 ] 85.91 \pm 0.21 ( 4.07 ) 87.03 \pm 0.83 ( 1.88 ) 94.33 \pm 0.14 ( 5.42 ) 6.20 \pm 0.08 ( 0.20 ) 140.06 SalUn [ 13 ] 89.89 \pm 0.25 ( 0.91 ) 89.23 \pm 0.21 ( 0.32 ) 97.71 \pm 0.08 ( 2.04 ) 7.20 \pm 0.13 ( 0.80 ) 32.77 SFRon [ 23 ] 90.58 \pm 0.28 ( 1.60 ) 86.69 \pm 0.79 ( 2.22 ) 99.02 \pm 0.01 ( 0.73 ) 13.20 \pm 1.25 ( 6.80 ) 114.39 RUM [ 47 ] 91.16 \pm 0.27 ( 2.18 ) 88.04 \pm 0.34 ( 0.87 ) 99.97 \pm 0.01 ( 0.22 ) 4.20 \pm 0.09 ( 2.00 ) 110.59 OUR 88.38 \pm 0.19 ( 0.60 ) 88.99 \pm 0.22 ( 0.08 ) 99.23 \pm 0.14 ( 0.52 ) 7.00 \pm 0.31 ( 0.60 ) 91.00 CIFAR20 - ViT Retrain 69.16 \pm 0.24 ( 0.00 ) 69.07 \pm 0.18 ( 0.00 ) 99.99 \pm 0.00 ( 0.00 ) 12.20 \pm 0.35 ( 0.00 ) 2313.38 FT [ 17 ] 64.00 \pm 0.21 ( 5.16 ) 68.50 \pm 0.25 ( 0.57 ) 94.29 \pm 0.34 ( 5.46 ) 13.80 \pm 1.35 ( 1.60 ) 96.42 GA [ 39 ] 61.21 \pm 0.15 ( 7.95 ) 74.77 \pm 0.23 ( 5.70 ) 80.66 \pm 2.72 ( 19.09 ) 24.80 \pm 6.34 ( 12.60 ) 146 RL [ 17 ] 63.69 \pm 0.27 ( 5.47 ) 67.48 \pm 0.13 ( 1.59 ) 88.45 \pm 0.31 ( 11.30 ) 13.80 \pm 1.25 ( 1.60 ) 71.29 IU [ 24 ] 62.21 \pm 0.15 ( 6.95 ) 82.24 \pm 0.24 ( 13.17 ) 86.05 \pm 0.76 ( 13.70 ) 31.40 \pm 3.58 ( 19.20 ) 57.41 FF [ 16 ] 60.19 \pm 0.08 ( 8.97 ) 88.24 \pm 0.13 ( 19.17 ) 88.10 \pm 0.45 ( 11.70 ) 33.24 \pm 4.53 ( 21.40 ) 1680.61 SCRUB [ 27 ] 62.72 \pm 0.21 ( 6.44 ) 73.51 \pm 1.43 ( 4.44 ) 76.58 \pm 1.65 ( 23.17 ) 17.60 \pm 1.32 ( 5.40 ) 194.61 l1 Sparse [ 24 ] 62.93 \pm 0.09 ( 6.23 ) 68.01 \pm 0.11 ( 1.06 ) 90.49 \pm 0.23 ( 9.26 ) 14.00 \pm 0.85 ( 1.80 ) 83.30 SalUn [ 13 ] 63.45 \pm 0.27 ( 5.71 ) 65.79 \pm 0.31 ( 3.28 ) 92.55 \pm 0.18 ( 7.20 ) 12.00 \pm 1.75 ( 0.20 ) 133.19 SFRon [ 23 ] 66.20 \pm 0.24 ( 2.96 ) 69.00 \pm 0.45 ( 0.07 ) 97.60 \pm 0.08 ( 2.15 ) 21.80 \pm 3.25 ( 9.60 ) 53.85 RUM [ 47 ] 64.87 \pm 0.18 ( 4.29 ) 67.13 \pm 0.24 ( 1.94 ) 91.36 \pm 0.43 ( 8.39 ) 11.00 \pm 0.76 ( 1.20 ) 103.10 OUR 67.90 \pm 0.21 ( 1.92 ) 69.37 \pm 0.19 ( 0.30 ) 97.72 \pm 0.08 ( 0.44 ) 12.80 \pm 1.24 ( 0.60 ) 187.56 Table 2 : Performance of Attacks on Class-wise Unlearning: Inferring the Membership of Unlearned Classes from OOD Classes Attacks Balanced Accuracy (TPR @0.1 FPR) / % Retrain FT GA RL IU BU SCRUB l1 Sparse SalUn SFRon RUM OUR CIFAR20 (-5) MIA-LiRA 52.28(31.25) 59.04(18.75) 61.57(37.50) 50.00(31.25) 64.70(18.75) 52.53(25.00) 50.42 (6.25) 50.00(25.00) 59.29(6.25) 89.02(87.50 ) 61.49(25.00) 52.45 (6.25) MIA-UP 50.00(18.75) 56.84(12.50) 64.70(37.50) 61.40(18.75) 69.00(50.00) 64.95(12.50) 50.00(0.00) 50.25(18.75) 67.40(50.00) 73.14(43.75) 50.00(18.75) 50.00(18.75) ReA 50.00(6.25) 86.99(75.00) 71.03(43.75) 95.52(93.75) 89.02(87.50) 91.05(93.75) 68.75(31.25) 63.34(25.00) 95.27(100.00) 84.29(81.25) 85.64(81.25) 52.28( 12.50 ) CIFAR100 (-5) MIA-LiRA 50.00(6.25) 57.52(6.25) 55.49(25.00) 69.00(25.00) 87.67(87.50) 81.42(75.00) 64.44(31.25) 59.54(18.75) 69.17(43.75) 65.37(37.50) 66.89(37.50) 50.00(6.50) MIA-UP 50.08(12.50) 63.60(43.75) 59.54(18.75) 56.59(25.00) 63.09 (25.00) 67.23(37.50) 52.70(6.25) 55.49(25.00) 64.27(37.50) 73.14(56.25) 54.05(6.25) 54.73(12.50) ReA 50.00(6.25)) 86.32(75.00) 60.22(31.25) 82.47(81.50) 89.24(89.75) 84.17(83.75) 61.57(31.25) 62.67(25.00) 81.33(62.50) 65.88(31.25) 82.77(87.50) 50.68(0.00) Table 3 : Performance of Attacks on Random Sample Forgetting (10%): Infer the Membership of Unlearned Samples Attacks Balanced Accuracy (TPR@0.1FPR) / % Retrain FT GA RL IU FF SCRUB l1 Sparse SalUn [ 13 ] SFRon RUM OUR CIFAR10 MIA-LiRA 50.63(6.40) 50.60(6.80) 51.50(11.00) 51.50(8.00) 53.60(12.20) 53.60(9.60) 50.90(8.40) 50.90(6.20) 52.40(7.20) 52.20(13.20) 51.00(4.20) 50.69(7.00) MIA-UP 49.99(9.80) 50.66(9.92) 52.72(15.40) 50.00 ( 10.24 ) 52.84(12.28) 52.50(12.12) 50.34( 9.72 ) 50.78( 9.96 ) 51.34( 11.44 ) 52.68( 14.36 ) 52.82(15.56) 50.70(9.88) ReA 50.30(7.00) 50.40(8.60) 55.60(15.60) 52.20 (9.60) 58.40(24.20) 57.90(15.00) 51.60 (6.00) 51.20 (6.00) 53.40 (6.80) 52.80 (13.60) 51.20(8.20) 50.48(7.20) CIFAR20 MIA-LiRA 51.20(12.20) 56.10(13.80) 61.80(24.80) 53.20( 13.80 ) 64.80(31.40) 68.20 (33.24) 57.60(17.60) 53.10( 14.00 ) 53.80( 12.00 ) 56.80(21.80) 52.10( 11.00 ) 53.20( 12.80 ) MIA-UP 51.39(9.24) 54.46(10.80) 57.62(12.88) 53.48(8.88) 60.14(12.92) 59.89(12.24) 56.04(11.52) 52.84(9.72) 52.02(8.00) 51.36(20.84) 51.36(8.96) 52.46(9.88) ReA 51.35(10.80) 57.00 ( 12.80 ) 72.90(44.00) 57.10 (12.20) 72.70(44.40) 75.30(45.20) 63.30(33.40) 57.60 (12.00) 55.40 (9.80) 61.70(28.00) 53.20 (10.00) 53.80 (11.50) (a) (b) (c) (d) (e) (f) Figure 3 : ROC Analysis of MIA (Selected Methods). 5.3 Quantified Analysis of Unlearning Residuals We quantify residuals in MU benchmarks to explain why ReA succeeds, and assess OUR’s effectiveness in scrubbing them. To validate this, we use distinct analyses for class-wise and sample-wise residuals. For class-wise cases, we visualize representation spaces to detect class-specific residuals. For sample-wise cases, we quantify residuals by measuring the recovered unlearned accuracy (UA) by ReA. Class-wise Residual Visualization . Figure 4 compares t-SNE [ 41 ] projections of unlearned models in CIFAR-20 (-5) experiments and quantifies them with representation metrics: (1) Overlap degree ( o ) for class separation, (2) Silhouette Score ( s ) for cluster compactness, and (3) Intra-class Variance ( \mathrm{var} ). OUR achieves near-retrained model alignment ( \Delta\mathrm{var}<3 vs. 352 for RUM [ 47 ] ), indicating effective residual scrubbing. Figure 5(a) further reveals a strong correlation (Pearson =0.88 ) between representation misalignment (1- \mathrm{ToW}(o,s,\mathrm{var}) ) and ReA accuracy, confirming that representation residuals enable privacy breaches. Full metric definitions and results are in Appendix E.6 . Sample-wise Residual Quantification . For sample-wise cases, we assess residuals by tracking unlearning accuracy recovery during ReA. Figure 5(b) shows that higher recovered accuracy ( \Delta\mathrm{UA} ) correlates with increased privacy leakage (Pearson =0.87 ). OUR minimizes both recovery ( \Delta\mathrm{UA}=1.26\% ) and privacy risk (ReA \leq 53.80\% ), while some methods like GA [ 39 ] exhibit strong recovery ( \Delta\text{UA}=14.85\% ) and high vulnerability (ReA \leq 72.90\% ). (a) (b) (c) Figure 4 : Representation space visualization of the unlearned model. OUR’s treatment for the fruit & vegetables forget class (red) matches retrained models better than RUM. Appendix E.6 provides results for all AMU methods. (a) (b) Figure 5 : Quantified Residuals versus ReA Attack Accuracy. 5.4 Analysis of OUR Variants To validate the necessity of OUR’s core components, we evaluate four ablated variants: (1) Orth w/o Replay (jointly optimizing orthogonal unlearning and utility preservation), (2) OUR w/o Sparsity (removing l1 regularization during replay), (3) RL w/ Replay (replacing orthogonal unlearning with random label (RL) [ 17 ] ), and (4) FT w/ Sparsity (fine-tuning only with l1 regularization [ 24 ] ). Figure 6 plots unlearning efficacy (measured by ToW(UA,TA,RA)) against ReA privacy attack accuracy across datasets, and three critical observations emerge. First, the two-phase design is essential. Orth w/o Replay performs the worst, which degrades unlearning efficacy by 0.06 and increases ReA attack accuracy to 95.52\% , showing that joint optimization in unlearning weakens both objectives. Second, orthogonal unlearning proves superior to alternatives. OUR reduces ReA accuracy by 2.89-6.12\% compared to RL w/ Replay , as it removes more internal residuals. Third, l1 regularization in OUR enhances class-wise unlearning (reducing ReA accuracy by 35.07\% ), yet has a limited effect on sample-wise cases where the separated replay phase dominates. The synergy of orthogonal unlearning, two-phase design and l1 regularization achieves both effective unlearning and privacy protection. (a) (b) Figure 6 : Unlearning performance of OUR variants versus their privacy vulnerability test by ReA. 5.5 Experiments on Image Generation We evaluate privacy risks in identity-sensitive face generation using SD models. The target model synthesizes two identities (e.g., “a photo of Laura”) with distinct features. Concept unlearning is performed to erase a target identity. Its unlearning efficacy is measured by Unlearning Identity Score Matching (ISM) [ 42 ] , abbreviated as UISM Retrained generation quality is assessed using three metrics: Retaining ISM ( RISM ) for identity accuracy, FID [ 13 ] for distribution fidelity, and BRISQUE [ 31 ] for artifacts. Appendix E.7 details setups and metrics and evaluates more unlearning benchmarks, e.g., SFRon [ 23 ] . As shown in Table 4 , OUR achieves near-retrained UISM with 12\% of retraining time but initially causes RISM degradation by 0.08 . To mitigate this, we integrate SalUn’s parameter freezing strategy to preserve retained identity features, improving RISM by 0.06 without compromising running-time efficiency. Visual results in Table 5 confirm that (SalUn-)OUR effectively removes target identities while maintaining retained output quality. We design two MIA to evaluate MU’s privacy leakage and report the results in Table 6 : (1) ReA , adapted from class-wise ReA, and (2) DiffAtk [ 46 ] , which employs adversarial prompts to revive forgotten identities within limited iterations. To be practical, attackers are assumed to have access to public photos of the forgotten identity (not original training data) and lack knowledge of the forgotten prompt. Both attacks run for 100 iterations, with success measured by the resonance index (Idx r ) where ISM exceeds 0.70 . ReA proves most effective, requiring 25 fewer Idx r in SalUn, while DiffAtk consistently fails. Second, orthogonal unlearning in OUR enhanced security for portrait generation compared to existing unlearning methods. OUR achieves a resilience index (Idx r ) of 70 (near-retrained models), even better than Meta-unlearning (Meta-Un) [ 14 ] , which is designed to prevent forgotten concept recovery. Table 4 : Performance of Identity-specific Concept Unlearning Metrics Retrain SalUn [ 13 ] Meta-Un [ 14 ] OUR SalUn-OUR UISM 0.13 \pm 0.02 ( 0.00 ) 0.49 \pm 0.03 ( 0.36 ) 0.37 \pm 0.05 ( 0.24 ) 0.16 \pm 0.02 ( 0.03 ) 0.26 \pm 0.01 ( 0.13 ) RISM 0.76 \pm 0.02 ( 0.00 ) 0.71 \pm 0.02 ( 0.05 ) 0.74 \pm 0.04 ( 0.02 ) 0.68 \pm 0.01 ( 0.08 ) 0.74 \pm 0.02 ( 0.02 ) FID( \downarrow ) 101.82 \pm 8.33 108.23 \pm 7.25 122.76 \pm 11.35 110.31 \pm 7.69 103.55 \pm 11.35 BRISQUE( \downarrow ) 10.76 \pm 1.38 15.15 \pm 0.41 15.02 \pm 2.03 15.58 \pm 3.90 12.03 \pm 3.82 RTE (s) 937.50 239.31 485.31 110.25 113.83 Table 5 : Images of Identity-specific Concept Unlearning Target Retrain SalUn [ 13 ] Meta-Un [ 14 ] OUR SalUn-OUR Unlearned Identity. Prompt: “ a photo of Sarah ” Retained Identity. Prompt: “ a photo of Laura ” Table 6 : Privacy Attack against Concept Unlearning Attacks Resonance Index, Idx r , (Maximum = 10) \times 10 @ ISM Retrain SalUn [ 13 ] Meta-Un [ 14 ] OUR SalUn-OUR DiffAtk [ 46 ] 10 \pm 0.0 @0.18 10 \pm 0.0 @0.10 10 \pm 0.0 @0.09 10 \pm 0.0 @0.10 10 \pm 0.0 @0.08 ReA 6.5 \pm 1.0 @0.71 2 \pm 0.5 @0.77 4.5 \pm 1 @0.71 7.5 \pm 1.0 @0.70 7 \pm 1 @0.72 6 Related Works Privacy Hazards in Machine Unlearning . The growing adoption of machine unlearning (MU) has raised concerns about its unintended privacy risks. Current research identifies two primary leakage sources: (1) updates between model versions before and after unlearning [ 9 , 21 , 5 ] , and (2) knowledge residuals caused by approximate unlearning algorithms [ 27 ] . While version control addresses the first, the second reveals more profound flaws: imperfect alignment between unlearned and test data creates measurable discrepancies. Crucially, we reveal that these discrepancies are not merely superficial; they manifest as residuals in the loss landscape. By exposing these hidden residuals, our work shows that existing unlearning methods inadvertently leak privacy. This highlights the need for stricter privacy guarantees beyond output-level evaluations. Forgotten Knowledge Recovery Phenomenon . Recent studies [ 34 , 14 , 22 ] on safety-aligned generative models have observed the Forgotten Knowledge Recovery Phenomenon (FRP), where supposedly removed harmful knowledge unexpectedly re-emerges through targeted relearning. This suggests that unlearned targets leave residual traces. Existing work [ 14 , 22 ] attributes these residuals to semantic/knowledge overlaps between harmful and other concepts. We go beyond this insights and examine FRP across a broad range of approximate machine unlearning (MU) methods [ 24 , 27 , 47 , 23 ] in closed-problem settings (with well-defined input/output spaces), and leverage FRP to uncover privacy risks. We find that algorithm-induced residuals persist regardless of semantic correlations , which poses measurable membership privacy threats. 7 Conclusion This paper reveals pervasive privacy vulnerabilities in approximate unlearning algorithms induced by implicit residuals. To exploit these residuals, we introduce Reminiscence Attack (ReA), a novel membership inference attack (MIA) that compromises unlearned data privacy. ReA extends its targets beyond unlearned samples to unlearned classes and concepts, posing a broader privacy threat. Compared to existing attacks like MIA-UP [ 27 ] relying on output-space residuals, ReA achieves up to 1.90\times higher attack accuracy. To mitigate this risk, we propose Orthogonal Unlearning & Replay (OUR), a framework that systematically scrubbing residuals to reduce privacy leakage. OUR generalizes across class-wise, sample-wise, and concept unlearning, achieving near-retrained unlearning efficacy while lowering privacy attack accuracy to near-random levels. Appendix Appendix A Algorithms In this section, we provide the pseudo-codes for our proposed privacy attack, the Reminiscence Attack (ReA) (described in Section 3 ), and the unlearning framework Orthogonal Unlearning & Replay (OUR) (defined in Section 4 ). Algorithm 1 Class-wise Reminiscence Attack (ReA) 1: Require : OOD dataset \mathcal{D}^{\text{ood}} , reference OOD set \mathcal{D}^{\prime} , assigned unlearned label y_{u} , loss function \mathcal{L} 2: Parameters : Max iterations \text{Idx}_{\text{max}} , learning rates \{\text{lr}_{j}\}_{j=1}^{J} 3: if Black-Box Scenario then 4: \theta^{\prime}_{0}\leftarrow\text{ModelExtract}(F(\cdot;\theta^{u})) { Model extraction } 5: F_{\text{ini}}\leftarrow\text{Copy}(F(\cdot;\theta^{\prime}_{0})) { Save initial model } 6: else if White-Box Scenario then 7: \theta^{\prime}_{0}\leftarrow\theta^{u} { Direct access } 8: end if 9: for \text{lr}_{j}\in\{\text{lr}_{j}\}_{j=1}^{J} do 10: Initialize \theta^{\prime}\leftarrow\theta^{\prime}_{0} 11: for i=1 to \text{Idx}_{\text{max}} do 12: Compute loss: L=\frac{1}{|\mathcal{D}^{\text{ood}}|}\sum_{x\in\mathcal{D}^{\text{ood}}}\mathcal{L}(F(x;\theta^{\prime}),y_{u})+\frac{1}{|\mathcal{D}^{\prime}|}\sum_{x\in\mathcal{D}^{\prime}}\mathcal{L}(F(x;\theta^{\prime}),F_{\text{ini}}(x)) 13: \theta^{\prime}\leftarrow\theta^{\prime}-\text{lr}_{j}\cdot\nabla_{\theta^{\prime}}L { SGD update } 14: if \text{Acc}_{\mathcal{D}^{\text{ood}}}(F(\cdot;\theta^{\prime}))>0.9 then 15: Record \text{Idx}_{\text{r}}(\mathcal{D}^{\text{ood}},\text{lr}_{j})\leftarrow i 16: break 17: end if 18: end for 19: end for 20: Compute confidence score: \mathcal{A}^{\prime}(\mathcal{D}^{\text{ood}})=1-\frac{1}{J\cdot\text{Idx}_{\text{max}}}\sum_{j=1}^{J}\text{Idx}_{\text{r}}(\mathcal{D}^{\text{ood}},\text{lr}_{j}) Equation 6 21: Output : Confidence score \mathcal{A}^{\prime}(\mathcal{D}^{\text{ood}}) Algorithm 2 Sample-wise Reminiscence Attack (ReA) 1: Require : Inference dataset \mathcal{D}_{\text{infer}} , victim model F(\cdot;\theta^{u}) { i.e., the unlearned model } 2: Parameters : Pseudo retain set size N_{r} , training epochs \text{EPOCHS}=5 , learning rate lr 3: Obtain confidence scores: \{s\}\leftarrow\mathcal{A}^{\prime}(\mathcal{D}_{\text{infer}}) { Using MIA-LiRA method } 4: Select top- N_{r} indices \{\text{ind}_{i}\}_{i=1}^{N_{r}} from \{s\} based on confidence 5: Construct pseudo retain set: \mathcal{D}_{r}^{\prime}\leftarrow\mathcal{D}_{\text{infer}}[\{\text{ind}_{i}\}_{i=1}^{N_{r}}] { Subset selection } 6: for e=1 to EPOCHS do 7: Compute loss: L=\frac{1}{|\mathcal{D}_{r}^{\prime}|}\sum_{(x,y)\in\mathcal{D}_{r}^{\prime}}\mathcal{L}(F(x;\theta^{\prime}),y) 8: \theta^{\prime}\leftarrow\theta^{\prime}-\text{lr}\cdot\nabla_{\theta^{\prime}}L { SGD update } 9: end for 10: Output : Updated confidence scores \{\mathcal{A}^{\prime}(F(x;\theta^{\prime}))\}_{x\in\mathcal{D}_{\text{infer}}} Algorithm 3 Orthogonal Unlearning & Replay (OUR) Framework 1: Require : Original model F(\cdot;\theta) , full dataset \mathcal{D} , unlearning set \mathcal{D}_{u} 2: Parameters : Phase epochs e_{1} , e_{2} , norm threshold \Delta_{\text{thr}} , learning rate lr 3: Compute remaining set: \mathcal{D}_{r}\leftarrow\mathcal{D}\setminus\mathcal{D}_{u} 4: Store initial parameters: \theta^{\text{ini}}\leftarrow\theta=[\theta_{1},\cdots] Phase 1: The Orthogonal Unlearning 1: for e=1 to e_{1} do 2: Compute orthogonal loss: L_{\text{orth}}=\sum_{(x,y)\in\mathcal{D}_{u}}\sum_{l=1}^{k}\|F_{l}(x;\theta)-F_{l}(x;\theta^{0})\|_{2}^{2} 3: \theta\leftarrow\theta-\text{lr}\cdot\nabla_{\theta}L_{\text{orth}} { Orthogonality Update } 4: Compute the maximum parameter change: \Delta_{\text{max}}\leftarrow\max_{i\in|\theta|}{\frac{\|\theta_{i}-\theta_{i}^{\text{ini}}\|_{2}}{|\theta_{i}|}} 5: if \Delta_{\text{max}}>\Delta_{\text{thr}} then 6: break { Early stopping } 7: end if 8: end for Phase 2: The Replay Phase 1: for e=1 to e_{2} do 2: Compute loss: L_{=}\frac{1}{|\mathcal{D}_{r}|}\sum_{(x,y)\in\mathcal{D}_{r}}\mathcal{L}(F(x;\theta),y) 3: \theta\leftarrow\theta-\text{lr}\cdot\nabla_{\theta}L { Retain knowledge } 4: end for 5: Output : Unlearned model F(\cdot;\theta^{u}) where \theta^{u}\leftarrow\theta Appendix B The Proportion of Stable Neurons after Orthogonal Unlearning Orthogonal unlearning induces rapid utility degradation by excluding the retained dataset, yet model functionality recovers efficiently in the replay phase due to minimal parameter perturbations. To validate this, we empirically analyze neuron stability in a ViT model trained on CIFAR-20 after class-wise unlearning. We define the change for each parameter as follows. Since parameters vary in size, we define the change of parameter \theta_{i} as: \small\Delta(\theta_{i})={\frac{\|\theta_{i}-\theta_{i}^{\text{ini}}\|_{2}}{|\theta_{i}|}}, (8) where \theta_{i}^{\text{ini}} represents the parameter’s state before unlearning. To account for differences in parameter dimensions, we normalize the change by its size |\theta_{i}| for better observation. The maximum parameter change is then defined as: \small\Delta_{\text{max}}=\max_{i\in|\theta|}{\frac{\|\theta_{i}-\theta_{i}^{\text{ini}}\|_{2}}{|\theta_{i}|}} (9) Figure 7 reports that parameter changes after unlearning remain highly localized, with magnitudes concentrated below 0.0005 , which is three times smaller than the most minor changes observed in random models. Further analysis in Figure 8 confirms that smaller parameter changes correlate strongly with faster utility recovery during replay, which requires fewer training epochs. These results demonstrate that orthogonal unlearning preserves critical neural patterns despite utility drops, enabling efficient restoration through targeted replay rather than full retraining. Figure 7 : Histograms of Parameter Changes. Figure 8 : Parameter Changes versus The replay speed measured by epochs. Appendix C Supplementary Visualized Loss Landscape We extend Figure 2 to CIFAR-10/100-ResNet18 in Figure 9 , confirming that residuals, i.e., discrepancies between unlearned and non-training data, consistently appear across architectures. These are reflected in fine-tuning trajectories (dashed lines): in class-wise unlearning (Figure 9(a) ), the loss of unlearned classes converges to sharp minima while OOD classes remain unconverged; in sample-wise unlearning (Figure 9(b) ), unlearned samples exhibit sharp loss drops while test losses stay stable. These differences expose membership privacy. (a) (b) Figure 9 : Visualized loss landscapes of unlearned data and non-training data around unlearned models in CIFAR100 and CIFAR10 experiments. Appendix D Experimental Setups and Approximate Unlearning Benchmarks D.1 Setups ReA Setups . For class-wise ReA attacks, the ratio of reference data for logits constraints to inferred data is set to 14:1 for CIFAR20 and 6:1 for CIFAR100 during reminiscence. The maximum number of training iterations \text{Idx}_{\text{max}} is set to 75 . We employ cross entropy loss \mathcal{L} during its reminiscence process, with SGD with momentum ( 0.9 ) and weight decay 5 e -4 . In sample-wise ReA, the size of “pseudo” retained dataset \mathcal{N}_{r} is 20,\!000 . OUR Unlearning Method Setups . For the implementation of OUR, we adopt the following hyper-parameter configurations. The training process consists of distinct phases tailored for different architectures. For ResNet18 experiments, the orthogonal unlearning and replay phases each span 8 epochs, with learning rates of 0.0018 and 0.005 , respectively; a 0.5 decay is applied at epoch 3 during replay phases. For ViT experiments, the unlearning phase lasts 7 epochs. The replay phase runs for 7 epochs for class-wise unlearning or 11 epochs for sample-wise unlearning, with learning rates of 0.0008 and 0.0004 , and decay factors of 0.5 and 0.2 at epochs 4 and 6 . The ablation study on the number of epochs in OUR is provided in Appendix E.2 . Besides, the regularization component involves an l_{1} regularization factor of 1 e -5 to enhance model sparsity [ 24 ] . Implementation Details . All classification experiments are conducted using PyTorch on 2 \times RTX 4090 GPUs, and diffusion model experiments are performed on 4 \times RTX 4090 GPUs. D.2 Approximate Machine Unlearning Benchmarks • Fine-tuning (FT) [ 44 , 24 ] . It leverages the phenomenon of catastrophic forgetting to continue training the pre-trained model F(.;\theta) on the remaining dataset \mathcal{D}_{r} for a limited number of iterations, to forget the unlearned dataset \mathcal{D}_{u} which is no longer trained upon. • Gradient Ascent (GA) [ 39 , 17 ] . This approach seeks to achieve unlearning by reversing the gradient descent process on \mathcal{D}_{u} , thereby erasing its optimization traces in the model F(.;\theta) . • Random Label (RL) [ 17 , 10 ] . It assigns random labels for \mathcal{D}_{u} to erase the model F(.;\theta) ’s memory of them. • Influence Unlearning(IU) [ 24 ] . It utilizes the woodfisher method. It estimates the influence of \mathcal{D}_{u} on F(.;\theta) and designs perturbation strategies to erase this influence from the parameters \theta . • FisherForgetting (FF) [ 16 ] . FF perturbs \theta with additive Gaussian noise, where the covariance is derived from the fourth root of the Fisher Information matrix over \mathcal{D}_{r} . While theoretically rigorous, its reliance on Fisher matrix inversion limits parallel efficiency and increases computational overhead compared to gradient-based methods. • Boundary-based Unlearning (BU) [ 8 ] . BU shifts the decision boundary of F(.;\theta) to mimic a retrained model’s behavior, thereby bypassing parameter-space optimization. Additionally, there are five optimized unlearning frameworks. • l1 sparsity [ 24 ] . This framework bridges approximate and exact unlearning by pruning non-critical weights. Sparsity reduces the parameter space, thus it narrows the gap between approximate and ideal unlearning outcomes while maintaining efficiency. • SalUn [ 13 ] . It introduces weight saliency to focus unlearning efforts on critical parameters. Analogous to input saliency in explainable AI, it prioritizes weights with high influence on \mathcal{D}_{u} . • SCRUB [ 27 ] . It uses a teacher-student architecture where the student selectively disregards the teacher’s knowledge about \mathcal{D}_{u} . This “unlearning-by-disobedience” approach scales without restrictive assumptions. • SFRon [ 23 ] . It unifies gradient-based MU by decomposing updates into three components: forgetting gradient ascent, retaining gradient descent, and saliency-guided weighting. It further incorporates a Hessian-aware manifold geometry to align unlearning trajectories with the output probability space to balancing utility performance and forgetting efficacy. • Refined-Unlearning Meta-algorithm (RUM) [ 47 ] . It refines the forget set \mathcal{D}_{u} into homogeneous subsets and applies specialized unlearning strategies to each. Its meta-algorithm orchestrates existing methods to comprehensively unlearn \mathcal{D}_{u} . Appendix E Complementary Experimental Results E.1 Ablation Study of Layers \{l\}_{k} Seleted in OUR To determine the optimal layer configuration for orthogonal unlearning (OUR), we conduct an ablation study evaluating three distinct layer-selection strategies for {l}_{k} : using output layers from the first three transformer blocks ( First 3 ), the last three blocks ( Last 3 ), and a distributed set comprising the first, middle, and final blocks ( Span 3 ). Experiments on CIFAR-20 evaluate unlearning performance through ToW of Unlearning and resistance against relearning attacks through ReA Accuracy, where values approaching 1.0 and 50\% respectively indicate optimal outcomes. As shown in Figure 10 , the Span 3 configuration achieves the most favorable balance across both evaluation metrics for class-wise and sample-wise OUR implementations, which attains near-optimal ToW while maintaining ReA accuracy closest to random-guess performance. Figure 10 : OUR performance with different \{l\}_{k} settings: first 3 / last 3 / span 3 denote the output layers of the first 3 blocks, last 3 blocks, and first, middle, and last blocks, respectively. E.2 Ablation Study of Number of Epochs in OUR To optimize the training efficiency and unlearning effectiveness of OUR unlearning, we conduct an ablation study examining epoch configurations for its two-phase training: orthogonal unlearning ( e1 ) and replay ( e2 ). We evaluate three critical metrics across five repeated trials on CIFAR20-ViT, CIFAR10-ResNet18, and CIFAR100-ResNet18: unlearning performance (ToW of Unlearning), and resistance against relearning attacks (ReA Attack Accuracy). As shown in Figure 11 , optimal configurations emerge for each setting: CIFAR20-ViT requires 2 e1 and 7 e2 for sample-wise and 7 e1 and 11 e2 for class-wise OUR; CIFAR10-ResNet18 requires 8 e1 and 8 e2 for sample-wise OUR; CIFAR100-ResNet18 requires 8 e1 and 8 e2 for class-wise OUR. These configurations simultaneously achieve near-optimal ToW ( \approx 1.0 ) and strong ReA resistance while minimizing RTE. (a) (b) (c) (d) (e) (f) (g) (h) Figure 11 : Ablation study of e_{1} and e_{2} in OUR. Gray vertical bars indicate error bars. Class-wise ReA is evaluated 10 times for both unlearn and OOD classes per experiment. Results are averaged over 5 independent trials. E.3 Ablation Study of ReA Learning Rate Set The learning rate (lr) during the reminiscence process largely impacts class-wise ReA efficacy, as it directly affects the scores ( \mathcal{A}^{\prime} in Equation 3 ) for membership detection. In contrast, sample-wise ReA remains robust to lr variations, requiring only a minimal lr of 0.1\times the training lr for model convergence. Here, we evaluate whether our multi-lr aggregation strategy (Section 3.1 ) enhances robustness to this parameter in class-wise ReA. Figure 12 compares resonance differences (the difference of resonance index between OOD and unlearned classes, \Delta Idx r ) for all MU methods at different lr on CIFAR-20, which is presented as bars. The left y-axis shows resonance differences \Delta Idx r (positive values indicate faster convergence for unlearned classes), averaged over 10 trials. The right y-axis contrasts ReA privacy attack accuracy under single-lr (blue lines) and multi-lr aggregation (red dashed line) strategies. Key observations. First, single lr =0.001 maximizes ReA effectiveness for most MU methods except GA [ 39 ] and its variant, SFRon [ 23 ] , which indicates that optimal lr ranges are dependent on MU methods. Second, ReA’s performance collapses at lr =0.1 . Third, the multi-lr aggregation strategy achieves superior results without method-specific tuning. Notably, our OUR method exhibits near-zero resonance difference, demonstrating inherent resistance to ReA attacks. Figure 12 : Resonance difference (Idx r of OOD classes minus that of unlearned classes) and ReA attack accuracy across learning rates in ReA. Gray bars represent error bars. E.4 Ablation Study of ReA Convergence Threshold To evaluate how convergence threshold affects ReA performance, we conduct an ablation study on this key parameter. The convergence threshold represents the prediction accuracy threshold in class-wise ReA to determine the resonance index. Experiments perform on CIFAR-20 and evaluate the ReA performance targeting five selected unlearning benchmarks: GA, RL, SCRUB, SalUn, and RUM. The results in Figure 13 confirm that thresholds between 70\% and 80\% yield optimal ReA attack accuracy across all evaluated unlearning methods. Figure 13 : ReA Attack Accuracy versus Convergence Threshold E.5 The Complete ROC Analysis of Privacy Attacks in Section 5.2 This section provides the full ROC analysis for machine unlearning benchmarks in CIFAR20 experiments. Figure 14 shows the ROC figures of class-wise membership inference attacks (MIA). Figure 15 shows the ROC figures of sample-wise membership inference attacks (MIA). (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Figure 14 : ROC of class-wise MIA. The notable fluctuations result from having only 300 inferred objects in each figure. (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Figure 15 : ROC of sample-wise MIA. E.6 The Complete Representation Visualization Analysis of Experiments in Section 5.3 This appendix provides additional visualizations of lower-dimensional embeddings for Figure 4 in Section 5.3 . All visualization results are presented in Figure 16 . Moreover, the representation attributes are quantified using the following metrics: • Intra-class Variance (Variance) (var) [ 18 ] : Measures class compactness by calculating the average squared distance from each point to the class centroid. A significantly lower variance than in the retrained model suggests that the unlearned data are densely clustered, indicating knowledge residue. • Silhouette Score ( s ) [ 37 ] : Assesses clustering quality through the average silhouette coefficient for each point within the unlearned class, ranging from -1 to 1. High scores suggest that the model retains familiarity with class knowledge, pointing to knowledge residue. • Overlap Degree (Overlap) ( o ) [ 2 ] : Evaluates the overlap between the unlearned subset and others using kernel density estimation (KDE). A lower score suggests an effective classification of unlearned data, indicating knowledge residue. (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Figure 16 : Representation space visualization of the class-wise unlearned model in CIFAR20 experiments. E.7 Supplementary Description of Setups and Experiments in Concept Unlearning E.7.1 Setups Models . We utilize the pretrained stable diffusion model stable-diffusion-v1-5 [ 35 ] with prior preservation enabled, fine-tuning both the text encoder and diffusion model for 50 epochs using AdamW optimizer (learning rate is 5 e -7 ), consistent with the work [ 42 ] . OUR Setups . For OUR unlearning method, the orthogonal unlearning is applied only to output layers of the last two hidden modular blocks ( up_blocks ) and the final convolutional layer ( conv_out ), with scaled orthogonal loss factors [ 5 e -9 , 5 e -8 ] to stabilize training. The two-phase optimization spans 10 epochs: 1 epoch for orthogonal unlearning (learning rate is 5 e -6 ) and 9 epochs for the replay phase (learning rate is 8 e -7 ), both using AdamW. Since the generation task involves fine-tuning a highly parameterized model, we do not apply L1 regularization to avoid compromising its capability. Concept-Unlearning Benchmark Setups . SalUn [ 13 ] utilizes relabeled prompts (e.g., “This is a photo of James”) to erase targeted concepts, while Meta Unlearning [ 14 ] hinders the reconstruction of unlearned concepts from related concepts during unlearning, which is described by “this is a photo of a woman”. In terms of parameter configuration, SalUn and SFRon adopt identical hyper-parameters with learning rates of 1 e -6 across 10 training epochs, whereas Meta Unlearning operates at a learning rate of 1 e -5 over 50 optimization steps. These settings strictly adhere to their original designs. Privacy Attack Setups . For ReA, distinct resonance indexes emerge between the unlearned and out-of-distribution (OOD) classes when fine-tuning with only the inferred class. Thus, ReA does not rely on retained or pseudo-retained datasets in concept unlearning. It employs a learning rate of 5 e -6 , and each inferred class consists of four samples. The prompt it used is a random name, e.g., Emily. For DiffAtk [ 46 ] , the number of adversarial tokens is set to 5 , and the embedding method for the adversarial prompt is prefix-based. E.7.2 Metrics The evaluation metrics for generated portrait outputs (defined in Section 5.5 ) are detailed as follows. • Identity Score Matching (ISM) [ 42 ] . It evaluates identity consistency between generated and reference faces using ArcFace embeddings [ 11 ] . Lower values indicate better identity preservation. For identity-specific concept unlearning, we introduce two metrics: Unlearning ISM (UISM) , which evaluates the ISM of the unlearned identity, and Retaining ISM (RISM) , which evaluates the ISM of the retained identity. Both are used to observe how effectively the unlearned identity is removed and the remaining identity is preserved. • Fréchet Inception Distance (FID) [ 25 ] . It quantifies the similarity between real and generated face distributions via Inception-v3 features. Lower scores reflect closer alignment to real data statistics. • BRISQUE [ 33 ] . It assesses perceptual quality without reference by detecting unnatural patterns (e.g., artifacts, textures) in spatial features. Lower values denote more natural outputs. • Resonance Index in ReA Attack (Idx r ). It measures the speed of identity recovery during the ReA process, which records the iteration where the ISM reaches 0.7 . Table 7 : Performance of Identity-specific Concept Unlearning Metrics UISM RISM FID( \downarrow ) BRISQUE( \downarrow ) RTE (s) Retrain 0.13 \pm 0.02 ( 0.00 ) 0.76 \pm 0.02 ( 0.00 ) 101.82 \pm 8.33 10.76 \pm 1.38 937.50 SFRon [ 23 ] 0.50 \pm 0.08 ( 0.37 ) 0.59 \pm 0.03 ( 0.17 ) 134.93 17.58 483.15 E.7.3 Experimental Results on an Additional Concept Unlearning Benchmark: SFRon [ 23 ] This section presents an evaluation of SFRon [ 23 ] as an additional concept unlearning benchmark. Table 7 shows its performance in identity-specific concept unlearning, where UISM remains as high as 0.50 , indicating ineffective removal of identity information. Table 9 provides further evidence, showing that the unlearning process mainly reduces image quality while preserving key identity features. As a result, SFRon is very vulnerable when attacked by ReA. It converges quickly in ReA, as shown in Table 8 , where its Resonance Index reaches only 10 , far below the retrained model’s 70 . Table 8 : Privacy Attack against Identity-specific Concept Unlearning Attacks Resonance Index, Idx r , (Maximum = 10) \times 10 @ ISM Retrain SFRon [ 23 ] DiffAtk [ 46 ] 10 \pm 0.0 @0.18 10 \pm 0.0 @-0.04 ReA 6.5 \pm 1.0 @0.71 4 \pm 0.5 @0.79 Table 9 : Images of Identity-specific Concept Unlearning Target Retrain SFRon [ 23 ] Unlearned Identity. Prompt: “ a photo of Sarah ” Retained Identity. Prompt: “ a photo of Laura ” (a) (b) (c) (d) Figure 17 : Training trajectories in the Reminiscence Attack under black-box scenarios. 3\% of each type of data is available. The light-colored areas represent the error bars at different learning rates. E.8 Evaluating Reminiscence Attack under Black-box Scenarios We investigate the class-wise ReA attack performance in black-box scenarios, to explore the transferability of knowledge residue from approximate unlearning in model extraction [ 40 ] . In white-box attacks, the attacker has direct access to the target model, allowing the ReA to be launched immediately. Conversely, in black-box scenarios, an additional preparatory step is required. This involves using data-free model extraction (ME) attacks (DFME) [ 40 ] to obtain a local substitute model on which the ReA is then launched. In black-box scenarios, we set the attacker’s query budget to 20 M and 30 M in the ME step for Cifar20 (-5) and Cifar100 (-5), around the primary setup of DFME [ 40 ] . Four approximate unlearning benchmarks are evaluated. Table 10 presents the accuracy and fidelity of substitute models obtained via DFME, and Table 11 reports the attack performance and computational cost of class-wise ReA against different unlearning methods. The adversarial set size is 3\% for CIFAR20 (-5) and 20\% for CIFAR100 (-5). Additionally, Figure 17 visualizes the reminiscence process in black-box settings, revealing that despite instability in training due to fidelity loss in ME, the unlearned dataset consistently exhibits distinct resonance. Results . Overall, ReA remains highly effective in black-box scenarios, nearly matching its white-box performance despite higher computational costs. This suggests that local models from ME inherit significant class-type residual knowledge , exposing fundamental vulnerabilities in approximate unlearning methods. Table 10 : Model Extraction Performance Dataset ME Accuracy (Fidelity) / \% FT GA RL IU Cifar20 (-5) 60.36 61.64 58.81 56.02 (77.73) (78.79) (73.65) (69.53) Cifar100 (-5) 54.83 56.74 58.81 56.29 (69.14) (71.24) (73.63) (70.54) Table 11 : Reminiscence Attack Under Black-Box Scenarios Dataset Accuracy (TPR @ 0.1 FPR) / \% Cost FT GA RL IU (s) Cifar20 (-5) 83.00 65.50 82.50 77.50 9.16e 3 (90.00) (55.00) (70.00) (80.00) Cifar100 (-5) 80.00 58.50 76.25 80.25 1.36e 4 (85.00) (40.00) (57.50) (65.00)