MUBox: A Critical Evaluation Framework of Deep Machine Unlearning Conference: SACMAT ’25: ACM Symposium on Access Control Models and Technologies; July 08–10, 2025; Stony Brook, NY CCS: Security and privacy Privacy protections CCS: Security and privacy Usability in security and privacy

Xiang Li, Bhavani Thuraisingham, Wenqi Wei · 2025

Read MUBox: A Critical Evaluation Framework of Deep Machine Unlearning Conference: SACMAT ’25: ACM Symposium on Access Control Models and Technologies; July 08–10, 2025; Stony Brook, NY CCS: Security and privacy Privacy protections CCS: Security and privacy Usability in security and privacy by Xiang Li, Bhavani Thuraisingham, Wenqi Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08576.txt.

Introduction

Xiang Li and co-authors build a test track for machine unlearning, the techniques meant to make a trained model forget specific data on request. Their 2025 paper presents MUBox, a platform integrating twenty-three unlearning methods, six practical scenarios, and eleven evaluation metrics, built because privacy law now demands forgetting that nobody can yet measure well.

Running the field through it produced uncomfortable findings. Even celebrated methods, including competition winners, worked inconsistently once the scenario moved past the easy settings most papers use. No single metric captured effectiveness, efficiency, and preserved usefulness together. Removing poisoned data varied wildly by attack type. The platform's lesson is metrological: a field cannot improve what it grades with a rubber ruler. MUBox offers a stiffer one, plus the comparative evidence to embarrass complacent methods into harder tests.

Catalog record

CREATORXiang Li, Bhavani Thuraisingham, Wenqi Wei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-08576

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MUBox: A Critical Evaluation Framework of Deep Machine Unlearning Conference: SACMAT ’25: ACM Symposium on Access Control Models and Technologies; July 08–10, 2025; Stony Brook, NY CCS: Security and privacy Privacy protections CCS: Security and privacy Usability in security and privacy Xiang Li, Bhavani Thuraisingham, Wenqi Wei Abstract . Recent legal frameworks have mandated the right to be forgotten , obligating the removal of specific data upon user requests. Machine Unlearning has emerged as a promising solution by selectively removing learned information from machine learning models. This paper presents MUBox, a comprehensive platform designed to evaluate unlearning methods in deep learning. MUBox integrates 23 advanced unlearning techniques, tested across six practical scenarios with 11 diverse evaluation metrics. It allows researchers and practitioners to (1) assess and compare the effectiveness of different machine unlearning methods across various scenarios; (2) examine the impact of current evaluation metrics on unlearning performance; and (3) conduct detailed comparative studies on machine unlearning in a unified framework. Leveraging MUBox, we systematically evaluate these unlearning methods in deep learning and uncover a set of key insights: (a) Even state-of-the-art unlearning methods, including those published in top-tier venues and winners of unlearning competitions, demonstrate inconsistent effectiveness across diverse scenarios. Prior research has predominantly focused on simplified settings, such as random forgetting and class-wise unlearning, highlighting the need for broader evaluations across more complex and realistic unlearning tasks. (b) Assessing unlearning performance remains a non-trivial problem, as no single evaluation metric can comprehensively capture the effectiveness, efficiency, and preservation of model utility. Our findings emphasize the necessity of employing multiple metrics to achieve a balanced and holistic assessment of unlearning methods. (c) In the context of depoisoning-removing the adverse effects of poisoned data-our evaluation reveals significant variability in the effectiveness of existing approaches, which is highly dependent on the specific type of poisoning attack. We believe MUBox will serve as a valuable benchmark, advancing research in machine unlearning and highlighting areas for future improvement. Codes are available at https://github.com/Jessegator/MUBox . 1. Introduction Machine learning (ML) models are trained on vast amounts of data that often contain sensitive, private, or copyrighted information. These models can inadvertently pose privacy risks by retaining and potentially exposing sensitive information from their training data ( Song and Mittal 2021 ; Salem et al. 2020 ; Shokri et al. 2017 ) . Moreover, recent legislation such as the General Data Protection Regulation (GDPR) in the European Union ( Mantelero 2013 ) , the California Consumer Privacy Act (CCPA) ( Harding et al. 2019 ) in the USA, the Personal Information Protection and Electronic Documents Act (PIPEDA) in Canada ( Office of the Privacy Commissioner of Canada 2018 ) , and the Personal Information Protection Law of China ( Chen and Sun 2021 ) mandate service providers to comply with users’ right to be forgotten ( Shastri et al. 2019 ) upon receiving the data removal requests. In response to increasing data privacy regulations, Machine Unlearning has emerged as a promising solution to selectively remove learned knowledge from ML models. This process enables ML models to forget specific data upon request, such as copyrighted data ( Li et al. 2024a ) and sensitive personally identifiable information (PII) data ( Layne 2024 ) . While retraining a model from scratch is a straightforward solution, it is computationally expensive and impractical for large-scale models. To address this challenge, researchers have developed various unlearning methods ( Guo et al. 2020 ; Baumhauer et al. 2022 ; Golatkar et al. 2021 ; Wu et al. 2020 ; Izzo et al. 2021 ; Neel et al. 2021 ; Bourtoule et al. 2021 ; Thudi et al. 2022 ; Heng and Soh 2024 ; Warnecke et al. 2023 ; Chundawat et al. 2023a ; Kurmanji et al. 2023 ; Mehta et al. 2022 ; Hoang et al. 2024 ; Golatkar et al. 2020b ; Golatkar et al. 2020a ; Foster et al. 2024b ; Foster et al. 2024a ; Chen et al. 2023 ; Jia et al. 2023 ; Tarun et al. 2023 ; Fan et al. 2024b ; Chundawat et al. 2023b ) , employing diverse strategies. Security researchers, practitioners, and service providers are confronted with a myriad of machine unlearning techniques available in the literature. However, there is a notable lack of comprehensive, unbiased evaluations, leaving a significant gap in the quantitative understanding of the strengths and limitations of these methods. First , existing evaluations often employ different model architectures and datasets, making it challenging to directly compare the effectiveness of various unlearning approaches. Second , existing evaluations are often limited to specific unlearning scenarios, such as random data forgetting, where a subset of training data from all classes is randomly selected to be unlearned. This randomness introduces bias, as certain data subsets may represent worst/best-case scenarios for unlearning, a factor not adequately addressed by current approaches. Third , existing evaluations often use different metrics, resulting in inconsistent and potentially biased comparisons that may favor certain methods. To advance the research on machine unlearning, it is essential to develop a comprehensive evaluation and analysis platform for rigorous assessment. However, constructing such a platform presents several key challenges: (1) ensuring a unified comparison of various unlearning methods using the same dataset and model configurations can be difficult due to differing data and model requirements; (2) incorporating a broad range of representative unlearning methods for comprehensive evaluation demands substantial effort, given the rapid evolution and diversity of unlearning techniques; (3) providing a diverse array of metrics to evaluate different unlearning methods under various unlearning scenarios requires consideration of multiple perspectives, e.g., efficiency and effectiveness of unlearning and accuracy of the unlearned model. To date, no existing work ( Grimes et al. 2024 ; Fan et al. 2024a ; Li et al. 2024b ; Xu et al. 2024 ) meets all the requirements. These studies either fail to provide practical guidance on the interaction between algorithms and implementations or overlook real-world applications and performance across diverse unlearning scenarios. To bridge this gap, we present MUBox, a framework designed to comprehensively evaluate and compare unlearning methods while fulfilling all the aforementioned essential criteria. MUBox enables researchers and practitioners to understand the effectiveness of various unlearning methods and conduct comparative studies across different unlearning scenarios in a uniform, comprehensive, informative, and extensible manner. Our contributions are as follows: • We thoroughly review machine unlearning papers published in top security, AI, and ML venues over the past five years (2019-2024). Among 49 deep machine unlearning papers, our reproducibility study reveals that 30.6% of them fail to include code for reproducing their experiments, and 50.0% of the provided code lacks detailed instructions or has no instructions at all. This highlights the urgent need for a benchmarking framework to enhance reproducibility, provide clear implementation guidelines, and foster trust in machine unlearning research. • We present MUBox, the first unified evaluation platform developed for machine unlearning , currently featuring 23 unlearning approaches in deep learning, 6 practical unlearning scenarios and 11 evaluation metrics for utility and efficacy criteria. To the best of our knowledge, MUBox offers the most extensive collection of unlearning methods and comprehensive evaluation metrics available to date . • Our investigation offers several key insights that

Read the complete text