Forget What’s Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning
Bihao Zhan, Jie Zhou, Junsong Li, Yutao Yang, Shilian Chen, Qianjun Pan · 2025
Read Forget What’s Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning by Bihao Zhan, Jie Zhou, Junsong Li, Yutao Yang, Shilian Chen, Qianjun Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12958.txt.
Introduction
Bihao Zhan and 11 co-authors propose a privacy method for continual learning in a 2025 paper whose title urges forgetting what is sensitive and remembering what matters. Continual learning models acquire knowledge sequentially, accumulating diverse information over time. Standard differential privacy applies a uniform budget, protecting all data equally and degrading model utility greatly.
Their framework first introduces token-level differential privacy, allocating privacy budgets adaptively based on the semantic sensitivity of single tokens. Private entities receive robust protection while general knowledge gets less added noise. A second module, privacy-guided memory sculpting, uses that sensitivity analysis to intelligently forget sensitive information from the model's memory. The aim is deployment of continual learning in privacy-sensitive areas without the usual steep utility loss.
Catalog record
| CREATOR | Bihao Zhan, Jie Zhou, Junsong Li, Yutao Yang, Shilian Chen, Qianjun Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-12958
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Forget What’s Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning Bihao Zhan, Jie Zhou, Junsong Li, Yutao Yang, Shilian Chen, Qianjun Pan Abstract Continual Learning (CL) models, while adept at sequential knowledge acquisition, face significant and often overlooked privacy challenges due to accumulating diverse information. Traditional privacy methods, like a uniform Differential Privacy (DP) budget, indiscriminately protect all data, leading to substantial model utility degradation and hindering CL deployment in privacy-sensitive areas. To overcome this, we propose a privacy-enhanced continual learning ( PeCL ) framework that forgets what’s sensitive and remembers what matters. Our approach first introduces a token-level dynamic Differential Privacy strategy that adaptively allocates privacy budgets based on the semantic sensitivity of individual tokens. This ensures robust protection for private entities while minimizing noise injection for non-sensitive, general knowledge. Second, we integrate a privacy-guided memory sculpting module. This module leverages the sensitivity analysis from our dynamic DP mechanism to intelligently forget sensitive information from the model’s memory and parameters, while explicitly preserving the task-invariant historical knowledge crucial for mitigating catastrophic forgetting. Extensive experiments show that PeCL achieves a superior balance between privacy preserving and model utility, outperforming baseline models by maintaining high accuracy on previous tasks while ensuring robust privacy 1 1 1 The code, datasets, and trained models will be made publicly available on GitHub. . Introduction The rapidly expanding field of Continual Learning (CL) seeks to enable models to acquire and integrate new knowledge sequentially without forgetting what they’ve already learned, much like human intelligence ( Yang et al. 2025 ; Shi et al. 2024 ) . This capability is essential for real-world applications where data streams are continuous and dynamic, such as in personalized recommendation systems, autonomous driving, and healthcare diagnostics. However, as CL models, particularly powerful Large Language Models (LLMs), continuously assimilate diverse and evolving datasets, they inherently accumulate vast amounts of information. Much of this information can contain sensitive personal or proprietary data ( Carlini et al. 2021 ; Charles et al. 2024 ) . This inherent characteristic poses significant and often overlooked privacy challenges, raising concerns about potential data leakage and misuse. Research shows that LLMs can inadvertently memorize sensitive or personally identifiable information (PII) during training or fine-tuning ( Meng et al. 2025 ; Kuo et al. 2025 ) . Once embedded, such data become extremely difficult to audit, modify, or erase, posing significant legal and ethical challenges ( Liu et al. 2025 ; Wang et al. 2024 ) . Differential Privacy (DP) offers a mathematically rigorous solution by limiting the influence of individual training samples on model outputs ( Dwork 2006 ) . While promising, applying DP to LLMs, especially in continual learning settings, is challenging. Standard approaches like DPSGD ( Abadi et al. 2016 ) and DP-FedAvg ( Takakura, Liew, and Hasegawa 2025 ) inject uniform noise into gradients or embeddings, often degrading performance in high-dimensional, diverse tasks. Moreover, they treat all data uniformly, ignoring the fine-grained nature of text where only specific tokens may require protection ( Flemings, Razaviyayn, and Annavaram 2024 ) . These limitations are exacerbated in batch-incremental continual learning, as data arrive sequentially in this setting, which most DP methods designed for static datasets and single-round training are ill-equipped to handle ( Abadi et al. 2016 ) . Specifically within CL, some works have investigated private data replay or private memory management to mitigate catastrophic forgetting while preserving privacy. In such scenarios, privacy risks can accumulate across training phases as each task introduces different types of sensitive information ( Desai et al. 2021 ) . Although recent methods such as synthetic data generation ( Murtaza et al. 2023 ) , privacy-aware memory modules ( Özdenizci, Rueckert, and Legenstein 2025 ; Mulrooney et al. 2025 ) , and auxiliary networks ( Kim et al. 2023 ) have been proposed, they often rely on strong assumptions including access to public data or task-specific architectures. Moreover, they generally lack support for selective forgetting to remove memorized sensitive content without degrading overall model performance ( Chourasia and Shah 2023 ) . This capability is essential for trustworthy and privacy-compliant continual learning. However, these methods often struggle to differentiate between the varying levels of sensitivity within data, leading to either insufficient protection for highly sensitive information or excessive noise injection for general knowledge. The primary challenges in achieving privacy-enhanced continual learning lie in three key areas. Firstly, effectively discerning and quantifying the sensitivity of different pieces of information at a fine-grained level (e.g., token-level in text) remains a significant hurdle. Without this nuanced understanding, a blanket privacy approach either over-protects non-sensitive data, leading to unnecessary utility loss, or under-protects truly sensitive data, resulting in privacy breaches. Secondly, balancing the often conflicting goals of privacy protection and knowledge retention (i.e., mitigating catastrophic forgetting) is a complex optimization problem. Injecting noise for privacy can inadvertently corrupt crucial historical knowledge, making it difficult for the model to recall past tasks. Finally, integrating a privacy mechanism seamlessly into the CL paradigm, where knowledge acquisition is sequential and dynamic, requires a novel architectural design that can adapt its privacy enforcement based on the evolving nature of the learned information. To address these challenges, we propose a novel privacy-enhanced continual learning ( PeCL ) framework that champions the principle of “forgetting what’s sensitive and remembering what matters”. Our approach is built upon two core innovations. First, we introduce a token-level dynamic Differential Privacy strategy that adaptively allocates privacy budgets based on the semantic sensitivity of individual tokens. This mechanism intelligently identifies and quantifies the privacy risk associated with each token, ensuring robust protection for private entities while minimizing noise injection for non-sensitive, general knowledge. Second, we integrate a privacy-guided memory sculpting module. This module leverages the fine-grained sensitivity analysis from our dynamic DP mechanism to intelligently and selectively forget sensitive information from the model’s memory and parameters. Crucially, it simultaneously and explicitly preserves the task-invariant historical knowledge that is vital for mitigating catastrophic forgetting. We conduct extensive experiments across various continual learning benchmarks, demonstrating that our method achieves a superior balance between privacy protection and model utility. Our approach significantly outperforms baseline models by maintaining high accuracy on previous tasks while ensuring robust privacy guarantees. In summary, our paper makes the following three main contributions: • We propose a novel token-level dynamic Differential Privacy strategy that adaptively allocates privacy budgets based on the semantic sensitivity of individual tokens, enabling fine-grained and efficient privacy protection in continual learning. • We introduce a privacy-guided memory sculpting module that intelligently forgets sensitive information from the model’s memory and parameters while explicitly preserving task-invariant historical knowledge crucial for mitigating catastrophic forgetting. • We empirically demonstrate that our framework achieves a superior balance between privacy protection and model utility, significantly outperforming state-of-the-art baselines in continual learning scenarios. Related Work Privacy-Preserving in Large Language Models The growing deployment of LLMs in privacy-sensitive domains has intensified interest in privacy-preserving training techniques. A key line of research applies DP to large-scale models, with early methods such as DPSGD ( Abadi et al. 2016 ) and recent advances like large-scale DP pretraining ( Yu et al. 2021 ) . Moreover, DP-MLM ( Meisenbacher et al. 2024 ) introduces per-token protection