Per-parameter Task Arithmetic for Unlearning in Large Language Models

Chengyi Cai, Zesheng Ye, Jiangchao Yao, Jianzhong Qi, Bo Han, Xiaolu Zhang, Feng Liu, Jun Zhou · 2026

Read Per-parameter Task Arithmetic for Unlearning in Large Language Models by Chengyi Cai, Zesheng Ye, Jiangchao Yao, Jianzhong Qi, Bo Han, Xiaolu Zhang, Feng Liu, Jun Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22030.txt.

Introduction

Chengyi Cai and co-authors refine a popular trick for making language models forget private information. Task arithmetic unlearns by computing a task vector, the change in parameters caused by tuning on the private data, and subtracting it from the model. The method is fast and simple.

Its flaw is over-forgetting: one blanket subtraction can damage parameters that other knowledge depends on. The team's 2026 preprint proposes per-parameter task arithmetic, scaling the subtraction parameter by parameter. Surgical forgetting matters because privacy requests, from personal data to copyrighted text, increasingly require removal that does not lobotomize the model. Finer control could make unlearning a routine operation rather than an emergency procedure.

Catalog record

CREATORChengyi Cai, Zesheng Ye, Jiangchao Yao, Jianzhong Qi, Bo Han, Xiaolu Zhang, Feng Liu, Jun Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22030

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Per-parameter Task Arithmetic for Unlearning in Large Language Models Chengyi Cai, Zesheng Ye, Jiangchao Yao, Jianzhong Qi, Bo Han, Xiaolu Zhang, Feng Liu, Jun Zhou Abstract In large language model (LLM) unlearning, private information is required to be removed. Task arithmetic unlearns by subtracting a specific task vector (TV)–defined as the parameter difference between a privacy-information-tuned model and the original model. While efficient, it can cause over-forgetting by disrupting parameters essential for retaining other information. Motivated by the observation that each parameter exhibits different importance for forgetting versus retention, we propose a per-parameter task arithmetic (PerTA) mechanism to rescale the TV, allowing per-parameter adjustment. These weights quantify the relative importance of each parameter for forgetting versus retention, estimated via grad ients (i.e., PerTA-grad) or the diagonal Fisher information approximation (i.e., PerTA-fisher). Moreover, we discuss the effectiveness of PerTA, extend it to a more general form, and provide further analysis. Extensive experiments demonstrate that PerTA consistently improves upon standard TV, and in many cases surpasses widely used training-based unlearning methods in both forgetting effectiveness and overall model utility. By retaining the efficiency of task arithmetic while mitigating over-forgetting, PerTA offers a principled and practical framework for LLM unlearning. 1 Introduction Large language models (LLMs) can continually acquire new knowledge ( lu2024spp ; luo2024wizardarena ) through post-training; however, the integration of newly ingested data may raise concerns regarding privacy, intellectual property, or misinformation ( karamolegkou2023copyright ) . Due to their tendency to memorize training data, LLMs may inadvertently disclose sensitive information when queried. LLM unlearning ( liu2025rethinking ; yao2024large ) aims to erase the memory of specified entities from LLMs to mitigate such risks, as shown in Figure 2 (a). Some training-based LLM unlearning methods achieve forgetting of specific entities (i.e., forget set) by designing carefully crafted unlearning loss functions ( NPO ; SimNPO ; yao2024large ; satimp ) and incorporating entities to be retained (i.e., retain set) to ensure that unrelated knowledge in the model remains unaffected ( GD ) . Another counterpart, task arithmetic , avoids multiple iterative training epochs with extensive data, as illustrated in Figure 2 (b), where full model and final model represent LLMs before and after unlearning, respectively. This approach achieves unlearning by subtracting from the full model a specific task vector (TV) ( TV ) for the forget set. TV denotes the parameter difference between a model finetuned solely on the forget set (hereafter referred to as the FgtOnly model) and the original pretrained model (hereafter the Origin model). Figure 1: The task of LLM unlearning and mainstream method categories. (a) depicts the problem setting, where the objective is to erase knowledge of specific entities. (b) contrasts training-based approaches with task arithmetic. Figure 2: Bottlenecks of task arithmetic methods. (a) illustrates that TV may steer the model toward the ascent direction of the retained gradient, leading to over-forgetting. (b) shows per-parameter divergence of TV–retain gradient relations, rendering the problem non-trivial and not solvable by a uniform weight. However, the potential correlation and coupling between the entity to be unlearned and other knowledge may cause the subtracted task vector to also contain changes in parameters crucial for preserving other knowledge, thereby risking excessive forgetting of entities that should be retained. Figure 2 (a) takes the task of unlearning 1% of entities from the TOFU ( tofu ) dataset as an example. The top 30 parameters with the largest values in the negated task vector (i.e., -V , which is added to the full model \theta_{\rm full} to obtain the final model \theta_{\rm final}=\theta_{\rm full}+(-V) ) were selected as examples. For these parameters, we plotted both the values in the negated TV and the gradient magnitudes with respect to the retain set (i.e., the retain gradient) of the same parameters. The results show that, for most of these parameters, the direction indicated by the negated TV aligns with the gradient ascent direction for the retain set. This implies that directly adding the negated TV to the full model would lead to forgetting of the entities that are supposed to be retained. A simple remedy is to add a uniform weight \omega\in\mathbb{R} satisfying 0<\omega<1 to TV to reduce the effect of TV (i.e., \theta_{\rm final}=\theta_{\rm full}+\omega\cdot(-V) ), thereby balancing between unlearning and retaining. However, as shown in Figure 2 (b), we find that such an approach may be suboptimal as it ignores per-parameter divergence. By plotting the negated TV and the retain gradient corresponding to different parameters, we observe that different parameters exhibit varying relations between TV and retain gradients, requiring a more sophisticated paradigm. After formulating the problem, in Section 3 , we propose the Per -parameter T ask A rithmetic (PerTA) mechanism as a solution to the aforementioned bottleneck. PerTA assigns different weights to each parameter in TV and performs a per-parameter multiplication (i.e., \theta_{\rm final}=\theta_{\rm full}+W\odot(-V) ) to flexibly control the magnitude of editing, where W is a matrix with the same size as \theta_{\rm full} . Parameters that are more pivotal for unlearning can be assigned higher weights, whereas those crucial for retention receive lower weights, aiming to facilitate both unlearning and retention. In Section 4 , we detail how per-parameter weights are estimated using absolute grad ients (which captures the importance of parameters given forget or retain sets, abbreviated as PerTA-grad) or the diagonal Fisher information approximation (which reflects the sensitivity of parameters to forget or retain sets, abbreviated as PerTA-fisher). We analyze its effectiveness by defining retain-forget ratio for parameters. We also extend PerTA to a generalized form and provide further discussion. In Section 5 , we evaluate PerTA on two commonly used unlearning benchmarks TOFU ( tofu ) and MUSE ( muse ) across multiple metrics. Results show that PerTA not only substantially outperforms its baseline, vanilla TV, but also exceeds the performance of several mainstream training-based unlearning methods. Training time analysis confirms that PerTA is efficient, while qualitative results illustrate its ability to retain knowledge upon effective unlearning. PerTA extends TV by preserving retention while enabling effective unlearning, all with high efficiency. Remarkably, it achieves performance surpassing several training-based unlearning methods, highlighting its practical effectiveness. Beyond empirical gains, PerTA offers a new task-arithmetic perspective for LLM unlearning research and introduces a flexible approach for balancing modification and retention in LLM task arithmetic. 2 Related Works LLM Unlearning. Machine unlearning ( unlearning1 ; unlearning2 ; unlearning3 ; unlearning4 ; lu2022quark ) aims to selectively remove some previously acquired knowledge from a model while preserving its overall utility. LLM unlearning has attracted increasing attention, playing a vital role in correcting misinformation, mitigating biases, and protecting privacy ( fantowards ; yao2024survey ; jang2022knowledge ) . Recent studies on LLM unlearning have advanced this field from multiple perspectives, including benchmarks ( tofu ; muse ; wmdp ) , frameworks ( openunlearning ) , evaluation protocols ( wangrethinking ; wangtowards ) , methodological innovations ( jia2024soul ; pawelczykcontext ; kadhe2024split ) , and hallucination mitigation ( shen2025lunar ; zhang2025rule ) . Different objectives and problem settings of unlearning are discussed in Appendix A.5 Among training-based unlearning methods, GA ( yao2024large ) is the pioneering work that minimizes the log-likelihood of the entities to be unlearned. GD ( GD ) improves it by incorporating the loss

Read the complete text