Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April

Zhaoyang Chu, Yao Wan, Zhikun Zhang, Di Wang, Zhou Yang, Hongyu Zhang, Pan Zhou, Xuanhua Shi, Hai Jin, David Lo · 2025

Read Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April by Zhaoyang Chu, Yao Wan, Zhikun Zhang, Di Wang, Zhou Yang, Hongyu Zhang, Pan Zhou, Xuanhua Shi, Hai Jin, David Lo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13755.txt.

Introduction

Zhaoyang Chu and nine co-authors ask whether code language models can be made to forget in a 2025 software security paper. Such models memorize sensitive training data and can repeat confidential code word for word when prompted. Fixes such as deduplication or differential privacy require retraining the whole model, which deployed systems can rarely afford.

The team studies machine unlearning, editing a trained model after the fact. They curated fifty thousand high-risk memorized samples as targets and compared two gradient-ascent unlearning methods. Their variant, CodeEraser, erases only the sensitive segments inside code while protecting the surrounding structure and function. Across three model families it removed targeted memorization effectively and cheaply while keeping the models useful.

Catalog record

CREATORZhaoyang Chu, Yao Wan, Zhikun Zhang, Di Wang, Zhou Yang, Hongyu Zhang, Pan Zhou, Xuanhua Shi, Hai Jin, David Lo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-13755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3744916.3764573 ISBN: 979-8-4007-2025-3/26/04 CCS: Software and its engineering Software development techniques CCS: Security and privacy Zhaoyang Chu, Yao Wan, Zhikun Zhang, Di Wang, Zhou Yang, Hongyu Zhang, Pan Zhou, Xuanhua Shi, Hai Jin, David Lo Abstract . While Code Language Models (CLMs) have demonstrated superior performance in software engineering tasks such as code generation and summarization, recent empirical studies reveal a critical privacy vulnerability: these models exhibit unintended memorization of sensitive training data, enabling verbatim reproduction of confidential information when specifically prompted. To address this issue, several approaches, including training data de-duplication and differential privacy augmentation, have been proposed. However, these methods require full-model retraining for deployed CLMs, which incurs substantial computational costs. In this paper, we aim to answer the following research question: Can sensitive information memorized by CLMs be erased effectively and efficiently? We conduct a pioneering investigation into erasing sensitive memorization in CLMs through machine unlearning—a post-hoc modification method that removes specific information from trained models without requiring full retraining. Specifically, we first quantify the memorization risks of sensitive data within CLM training datasets and curate a high-risk dataset of 50,000 sensitive memorized samples as unlearning targets. We study two widely used gradient ascent-based unlearning approaches: the vanilla and constraint-based methods, and introduce CodeEraser , an advanced variant that selectively unlearns sensitive memorized segments in code while preserving the structural integrity and functional correctness of the surrounding code. Extensive experiments on three families of CLMs, i.e. , CodeParrot, CodeGen-Mono, and Qwen2.5-Coder, validate the effectiveness and efficiency of CodeEraser in erasing targeted sensitive memorization while maintaining model utility. 1. Introduction Recently, Code Language Models (CLMs), such as CodeGen ( Nijkamp et al. 2023 ) , Code Llama ( Rozière et al. 2023 ) , and Qwen2.5-Coder ( Hui et al. 2024 ) , have demonstrated significant potential in automating various aspects of software engineering, including code generation ( Du et al. 2024 ; Jiang et al. 2024 ; Wang et al. 2025a ) , code summarization ( Al-Kaswan et al. 2023 ; Arakelyan et al. 2023 ; Wan et al. 2018 ) , program repair ( Xia et al. 2023 ; Xia and Zhang 2024 ) , and type inference ( Li et al. 2025 ) . Their success can be attributed to pre-training with autoregressive language modeling on large-scale code corpora ( Wan et al. 2022 ; Wan et al. 2024 ) , where the model predicts the next token given a sequence of previous tokens. Figure 1 . An illustration of existing methods, i.e. , (a) data de-duplication, (b) differential privacy, and (c) machine unlearning, to mitigate the memorization of sensitive information in CLMs. We mask the personal details for ethical considerations. However, studies have shown that the current pre-training paradigm may retain sensitive data, e.g. , emails and passwords, encountered during the training phase ( Al-Kaswan et al. 2024 ; Carlini et al. 2021 ; Carlini et al. 2023 ; Niu et al. 2023 ; Yang et al. 2024 ) . This retention occurs because CLMs, trained on vast amounts of code collected from GitHub repositories, may inadvertently memorize sensitive data embedded within these repositories, including personally identifiable information ( e.g. , names , emails , and phone numbers ) and private authentication credentials ( e.g. , passwords , API keys , and cryptographic secrets ) ( Basak et al. 2023 ; Feng et al. 2022 ; Huang et al. 2024 ; Meli et al. 2019 ; Niu et al. 2023 ; Yang et al. 2024 ) . From another perspective, to strengthen individual control over personal data, global legislative frameworks such as the European Union’s General Data Protection Regulation (GDPR) ( Union 2018 ) and the California Consumer Privacy Act (CCPA) ( of California 2023 ) have established the “ Right to Be Forgotten ” (RTBF) ( Veale et al. 2018 ; Villaronga et al. 2018 ) . These regulations empower individuals to request the deletion of their personal data, providing a critical safeguard for privacy protection. To mitigate the privacy risks posed by potential data leaks in CLMs and ensure compliance with the RTBF, we investigate the following question: Can sensitive information memorized by CLMs be erased effectively and efficiently? Intuitive Approaches and Limitations. Our investigation reveals two distinct research directions. The first focuses on training data de-duplication, as illustrated in Figure 1 (a). Prior studies ( Carlini et al. 2023 ; Kandpal et al. 2022 ; Lee et al. 2022 ) have demonstrated that de-duplication can mitigate the memorization tendencies of LMs. However, experimental evidence indicates that LMs still retain substantial memorization capabilities even under this paradigm ( Biderman et al. 2023a ; Biderman et al. 2023b ; Jang et al. 2023 ) . Another line of research falls into Differential Privacy (DP) ( Abadi et al. 2016 ; Zhu et al. 2022 ) , as shown in Figure 1 (b). This approach enforces the formal guarantee that the addition or removal of any training data point does not substantially affect the final model ( Zhu et al. 2017 ) , thereby providing formal privacy guarantees for individual training samples. However, DP-based training fundamentally limits LMs’ ability to capture long-tail patterns in data distributions, resulting in significant utility degradation ( Anil et al. 2022 ; Feldman 2020 ; Feldman and Zhang 2020 ) . Furthermore, both DP and de-duplication methods are typically applied during the initial training phase. For already deployed CLMs, these methods lack the ability to selectively remove specific data as requested by users, often necessitating retraining the entire model ( Chen and Yang 2023 ; Jang et al. 2023 ; Nguyen et al. 2025 ) . Such retraining is costly and time-consuming, especially given the escalating scale of contemporary CLMs. This limitation prevents their flexibility in addressing dynamic user requests and evolving privacy demands in real-world scenarios. Our Work: A Machine Unlearning Perspective. More recently, machine unlearning has emerged as a promising alternative for LMs, as shown in Figure 1 (c), which seeks to remove specific information by post-hoc modifying the trained model ( Bourtoule et al. 2021 ; Cao and Yang 2015 ; Chen and Yang 2023 ; Golatkar et al. 2020 ; Jang et al. 2023 ) . Existing approaches typically employ gradient ascent to reverse the learning of specific data, thus proactively removing its influence ( Chen and Yang 2023 ; Jang et al. 2023 ) . Compared to DP and de-duplication techniques, machine unlearning enables LMs to quickly forget certain information with just a few parameter updates without full retraining, thereby reducing the training time from 900 \sim 1800 to \sim 0.001 A100 GPU days ( Jang et al. 2023 ) . Nevertheless, we argue that these approaches often indiscriminately forget entire text instances rather than selectively targeting specific sensitive information. As a result, they struggle to erase sensitive segments ( e.g. , API key strings) embedded in code without disrupting the structural integrity and functional correctness of

Read the complete text