ICCU: In-Context Continual Unlearning via Pattern-Induced Refusal Rules

Ruihao Pan, Suhang Wang · 2026

Read ICCU: In-Context Continual Unlearning via Pattern-Induced Refusal Rules by Ruihao Pan, Suhang Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27138.txt.

Introduction

Pan and Wang propose ICCU, an in-context ongoing unlearning method. It induces readable refusal rules from unlearning datasets. It applies them at run time as a filter or via the system prompt, without changing model weights. Since rules gather as an order-free union, the method is modular and free of cross-request clashes. The source forget-set data can be dropped after rule mining. Tests show ICCU blocks target knowledge while keeping utility, scales across serial requests, and stays robust to rephrased and cross-lingual queries.

Catalog record

CREATORRuihao Pan, Suhang Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82521
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27138

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ICCU: In-Context Continual Unlearning via Pattern-Induced Refusal Rules Ruihao Pan, Suhang Wang Abstract Machine unlearning aims to remove the influence of specific data from trained language models. In real-world deployments, unlearning requests often arrive sequentially, which challenges existing fine-tuning-based methods: fine-tuning each request is costly, accumulates utility loss, and may cause cross-request interference. To address these issues, we propose ICCU ( I n- C ontext C ontinual U nlearning), an in-context continual unlearning framework that induces readable refusal rules from unlearning datasets and applies them at inference time either as a filter or via the system prompt, without modifying model parameters. Because rules are accumulated as an order-independent union, ICCU is compositional and free of cross-request interference, and the original forget-set data can be discarded after rule induction. Extensive experiments show that ICCU effectively suppresses target knowledge while preserving utility, scales across sequential requests, and remains robust to paraphrased and cross-lingual queries. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities on various tasks ( 12 ; 42 ) . However, they are trained on web-scale corpora that inevitably contain sensitive, harmful, or copyrighted information, which LLMs could memorize and reproduce ( 5 ; 19 ; 20 ) . Such content often needs to be removed from the trained LLMs to comply with privacy regulations such as the GDPR’s “right to be forgotten” ( 22 ) , to address copyright disputes ( 14 ) , or to mitigate the spread of hazardous knowledge ( 20 ) . As retraining from scratch is prohibitively expensive, machine unlearning , which aims to effectively eliminate the influence of specific data from trained models, has attracted increasing attention ( 4 ; 26 ; 43 ) . Existing unlearning approaches largely rely on fine-tuning or parameter updates ( 18 ; 21 ; 20 ; 45 ) , which, while effective on a single request, are costly and difficult to apply repeatedly on large models ( 26 ; 30 ) . In practice, deletion requests may keep emerging over time: individuals, copyright holders, and content moderators may issue them at any point throughout a model’s deployment lifecycle ( 22 ; 14 ; 20 ; 11 ) . This sequential arrival of requests gives rise to the problem of continual unlearning ( 11 ; 40 ; 41 ) . A natural approach to continual unlearning is to apply an existing fine-tuning-based unlearning method ( 20 ; 18 ; 45 ; 21 ) to each incoming request. However, applying such methods sequentially raises three challenges: (i) cost —each round of fine-tuning is computationally expensive, and this cost accumulates over sequential requests ( 25 ; 26 ) ; (ii) relearning —previously removed knowledge may reappear after subsequent rounds, undoing earlier unlearning ( 11 ; 41 ) ; and (iii) utility erosion —the model’s general capability tends to degrade as more rounds of fine-tuning are applied ( 40 ; 41 ) . Very few recent works are published on continual unlearning (Section 2 ), and most of them are still fine-tuning-based and thus inherit the same challenges. Hence, continual unlearning remains underexplored, which motivates us to design an efficient and effective framework that can simultaneously tackle the above issues. To address these challenges, we propose ICCU, a framework that performs continual unlearning via in-context unlearning without modifying model parameters. For each unlearning request, ICCU clusters its samples into semantic groups and prompts an LLM to induce high-level patterns, which are converted into readable refusal rules. Sequential requests are handled by simply accumulating their rules into a shared repository, enabling scalable unlearning without parameter updates or cross-request interference. At inference time, ICCU dynamically selects a relevant subset of rules to regulate model behavior, supporting two complementary deployment modes: (i) filter-based unlearning (Section 6 ), where ICCU acts as a standalone semantic filter that intercepts forget-set queries with a separate refusal decision before generation, and (ii) end-to-end unlearning (Section 7 ), where rules are injected into the system prompt so that filtering and answering are jointly handled in a single LLM call. The two modes share the same rule induction pipeline but differ in their cost–interface trade-off: the filter-based mode runs refusal and generation as two separate calls and offers a clean, modular interface, while the end-to-end mode merges them into a single call, reducing inference cost while achieving comparable effectiveness. A single rule set can thus be reused across deployment scenarios with different latency and integration constraints. ICCU offers five key advantages over existing approaches. First , it is training-free and continual by design : new requests are handled by appending rules rather than retraining, eliminating cross-request interference and making the final behavior invariant to request order. Second , it is privacy-friendly : once rules are induced, the forget-set data is discarded, and only cluster centroids and rules—from which raw samples cannot be recovered—are kept. Third , it is semantically robust : since matching relies on semantic representations rather than surface-level lexical patterns, ICCU generalizes to paraphrased and many cross-lingual queries, where keyword- or classifier-based methods often degrade sharply. Fourth , it is readable and controllable : each rule is a human-readable description of an unlearning-related topic, making the policy auditable and controllable at the request or rule level. Fifth , it is flexibly deployable : rules can be freely composed and selectively activated at inference time, letting one rule repository serve different deployment scenarios. Our contributions are: (i) we study a new problem of in-context based continual unlearning; (ii) we propose ICCU , a novel in-context, rule-based framework that performs continual unlearning without modifying model parameters, inducing compact, readable refusal rules that are applied in either filter-based (Section 6 ) or end-to-end (Section 7 ) mode and are free of cross-request interference; and (iii) extensive experiments demonstrate the effectiveness of ICCU on continual unlearning. 2 Related Work LLM unlearning aims to eliminate the influence of specific data or knowledge from a trained LLM without retraining from scratch ( 43 ; 31 ) . Existing approaches can be broadly divided into parameter-updating and parameter-free methods ( 26 ; 39 ; 30 ) . Parameter-updating approaches modify model weights to suppress or overwrite unwanted knowledge, either through gradient-based optimization ( 18 ; 43 ; 45 ; 21 ; 23 ; 10 ; 35 ) or by altering internal representations ( 29 ; 20 ; 9 ; 32 ; 38 ; 13 ) . However, such methods often suppress behaviors rather than remove the underlying knowledge, leaving it vulnerable to relearning ( 27 ; 17 ) . Parameter-free approaches instead control model behavior through inference-time mechanisms without modifying parameters ( 34 ; 28 ; 33 ; 37 ; 25 ; 36 ; 3 ) . Continual unlearning, which targets sequentially arriving deletion requests, has recently drawn increasing attention, and most existing methods for it are fine-tuning-based ( 11 ; 6 ; 44 ; 40 ; 37 ; 41 ) . An extended discussion of in-context unlearning and continual unlearning is in Appendix A . 3 Problem Definition We study unlearning in a continual setting, where deletion requests arrive sequentially over a model’s deployment lifecycle rather than all at once. In practice, the knowledge to be removed is rarely isolated: forgetting the target

Read the complete text