CaTNiP : LLM Unlearning via Calibrated and Tokenized Negative Preference Alignment Zhengbang Yang, Yisheng Zhong, Junyuan Hong, Zhuangdi Zhu Abstract Pretrained knowledge memorized in LLMs raises critical concerns over safety and privacy, which has motivated LLM Unlearning as a technique for selectively removing the influences of undesirable knowledge. Existing approaches, rooted in Gradient Ascent (GA), often degrade general domain knowledge while relying on retention data or curated contrastive pairs, which can be either impractical or data and computationally prohibitive. Negative Preference Alignment has been explored for unlearning to tackle the limitations of GA, which, however, remains confined by its choice of reference model and shows undermined performance in realistic data settings. These limitations raise two key questions: i) Can we achieve effective unlearning that quantifies model confidence in undesirable knowledge and uses it to calibrate gradient updates more precisely, thus reducing catastrophic forgetting? ii) Can we make unlearning robust to data scarcity and length variation? We answer both questions affirmatively with CaTNiP (Calibrated and Tokenized Negative Preference Alignment), a principled method that rescales unlearning effects in proportion to the model’s token-level confidence, thus ensuring fine-grained control over forgetting. Extensive evaluations on MUSE and WMDP benchmarks demonstrated that our work enables effective unlearning without requiring retention data or contrastive unlearning response pairs, with stronger knowledge forgetting and preservation tradeoffs than state-of-the-art methods. \workshoptitle Socially Responsible and Trustworthy Foundation Models 1 Introduction Large Language Models are disruptive technologies built upon vast accumulations of human knowledge [ 34 ] . While their unprecedented capabilities have benefited society across various domains [ 2 , 27 , 41 ] , the massive pretrained knowledge memorized in LLMs poses a double-edged challenge, which raises concerns over safety, privacy, and intellectual property [ 6 , 5 ] . LLMs may inadvertently surface hazardous procedural information [ 29 ] , copyrighted books [ 42 , 8 ] , or sensitive personal data memorized during pretraining [ 6 , 20 ] that violate regulatory requirements [ 11 ] or ethical norms. Towards removing undesirable knowledge from LLMs, retraining from scratch [ 4 , 45 ] offers an oracle-level solution, which is prohibitively costly and even infeasible. Instead, a growing field of work explores LLM unlearning [ 50 , 42 , 8 , 29 ] , a methodology that selectively mitigates the influences of undesirable knowledge, as a more practical path towards accountable LLMs. At the core of varying LLM unlearning approaches is Gradient Ascent (GA) [ 22 , 49 ] , which fine-tunes a target LLM by increasing the loss gradient on data representing the undesirable knowledge, named unlearning data to weaken its influence. However, GA introduces a fundamental tradeoff that, while removing harmful knowledge, it also risks degrading general-domain knowledge, due to the interconnected nature of pretrained knowledge within LLMs, whereas GA uniformly increases the model’s predictive loss on forgetting data regardless of the semantic importance of data samples. Towards addressing this unlearning-preserving tradeoff , previous work often hinges on access to a subset of pretraining data, termed retention data , for preserving general domain knowledge during unlearning optimization, which could be a strong prerequisite in practice. Another line of research tackles the catastrophic collapse caused by GA objectives, among which Negative Preference Optimization (NPO) is a representative method [ 50 ] . NPO takes inspiration from LLM alignment objectives that initially required contrastive pairs (desired vs. undesirable responses) [ 39 , 35 ] . NPO relaxes this data requirement and instead optimizes only the tractable component tied to undesirable responses ( i.e. knowledge to be forgotten), making it more suitable for knowledge embedded in large corpora, such as copyrighted books. NPO still shows empirical limitations in unlearning efficacy and usually requires retention data to achieve more balanced performance [ 42 ] . The limitations may be rooted in its choices of alignment objectives, where a reference model is critical to indicate the margin for the unlearning model to improve [ 32 ] , which is reflected in the probability ratio between the unlearning model \pi_{\bm{\theta}} and a reference model \pi_{\text{ref}} given an unlearning sample (x,y) : \frac{\pi_{\bm{\theta}}(y|x)}{\pi_{\text{ref}}(y|x)} . Prior work typically uses a static reference model \pi_{\text{ref}} fixed at initialization, e.g. model before alignment, which offers limited margin to guide the unlearning model, especially in regions where \pi_{\text{ref}}(y|x) is already high, which leads to diminished unlearning guidance as training progresses. Furthermore, the varying unlearning samples introduce training biases, as long samples contribute more to gradient updates regardless of their semantic importance. This mismatch is exacerbated when evaluation data follow diverging length distributions that are different from those seen in training, which further hinders unlearning and alignment efficacy [ 26 ] . Towards overcoming the limitations of prior arts, we focus on addressing two key questions: i ) How to achieve effective unlearning with an informative reference model , that can guide model gradient update more effectively and precisely, while avoiding catastrophic forgetting without relying on retention data? ii ) how to make unlearning robust to data length bias, while benefiting from heterogeneous or scarce unlearning data, such as concept unlearning with only a few anchor examples [ 43 ] ? In response, we proposed CaTNiP , an unlearning algorithm based on Ca liberated and T okenized N egat i ve P reference Alignment. Our innovation lies in the unlearning objective design to capture the heterogeneous influence of tokens on the unlearning process. We introduced a calibrated objective by re-weighting each loss term based on an adaptive reference model , which rescales the unlearning effects in proportion to the model’s predictive confidence. In parallel, our objective is tokenized such that each token independently contributes to the unlearning loss, which provides fine-grained unlearning optimization that focuses on a token’s semantic importance, while remaining robust to training biases induced by varying data lengths. Overall, we introduced an effective unlearning method with calibrated, token-level alignment based on the model’s prior confidence in the unlearning knowledge. We verified the key factors in our algorithm design that enhance its unlearning outcomes, including the choice of reference policy, calibration gradient, effects of tokenization, and its performance robustness against varying qualities of training data and task context. CaTNiP offers a principled solution that enables effective unlearning without requiring retention data or curating contrastive unlearning response pairs , while achieving comparable or stronger tradeoffs between forgetting and knowledge preservation than state-of-the-art unlearning methods. 2 Preliminaries of Unlearning We consider an LLM as a policy model \pi_{\bm{\theta}} parameterized as {\bm{\theta}} , which contains undesirable knowledge manifested in an unlearning dataset {\mathcal{D}} . Each unlearning sample \tau=(x,y)\sim{\mathcal{D}} contains input x and undesirable response y . The goal of LLM unlearning is to reduce model’s knowledge of {\mathcal{D}} while preserving the general-domain knowledge, which is typically summarized as below: \displaystyle\min_{\bm{\theta}}{\mathcal{L}}({\bm{\theta}})={\mathcal{L}}_{\text{unlearn}}({\bm{\theta}};{\mathcal{D}})+{\mathcal{L}}_{\text{retain}}({\bm{\theta}};{\mathcal{D}}_{\text{retain}}),\vskip-21.68121pt where {\mathcal{D}}_{\text{retain}} denotes a dataset of general domain knowledge intended to be preserved, termed the retaining dataset, which may not always be available during unlearning in practice, due to the prohibitive cost of data processing or restricted permission. Among varying formulations for the {\mathcal{L}}_{\text{unlearn}} loss, Gradient Ascent (GA) is a fundamental building block, which minimizes the log probability for the model to generate the undesirable response: \min_{\bm{\theta}}{\mathcal{L}}^{\text{GA}}_{\text{unlearn}}({\bm{\theta}};{\mathcal{D}})=\mathbb{E}_{x,y\sim{\mathcal{D}}}[\log\pi_{\bm{\theta}}(y|x)]. The core challenge of effective unlearning is to keep a balanced performance between forgetting and knowledge retention. Prior unlearning work typically relies on access to {\mathcal{D}}_{\text{retain}} during training and makes the retain loss tractable by minimizing the behavior difference on the {\mathcal{D}}_{\text{retain}} between the target model {\bm{\theta}} and a reference model, which is usually the model before unlearning training. For instance, a widely used formulation employs the KL divergence [ 31 ] : \displaystyle\min_{\bm{\theta}}{\mathcal{L}}^{\text{KL}}_{\text{retain}}({\bm{\theta}};{\mathcal{D}}_{\text{retain}})=\mathbb{E}_{x\sim{\mathcal{D}}_{\text{retain}}}\Big[\mathbb{D}_{\text{KL}}[\pi_{\bm{\theta}}(\cdot|x)\|\pi_{\text{ref}}(\cdot|x)]\Big]. (1) 2.1 LLM Unlearning As Preference Optimization Unlearning is also closely connected to LLM Alignment , which is a paradigm to optimize the LLM’s preference over responses to align with those of humans. A representative method along this line is Direct Preference Optimization (DPO) [ 39 ] . Formally, when given a pair of preferred and less preferred model responses, \tau^{+}=(x,y^{+}),\tau^{-}=(x,y^{-}) towards the same input x , an alignment optimization maximizes the relative probability for model \pi_{\bm{\theta}} to generate the desirable response over the less desirable one: \displaystyle\min_{\pi_{\bm{\theta}}}\mathbb{E}_{(\tau^{+},\tau^{-})\sim\mathcal{D}}\Big\{-\log P(\tau^{+}\succ\tau^{-}|\pi_{{\bm{\theta}}})\Big\}. (2) DPO treated the above as a constrained RL optimization task and reformulated the objective to be reward-free: \displaystyle{\mathcal{L}}_{\text{DPO}}=-\frac{1}{\beta}\mathbb{E}_{(x,y^{+},y^{-})\sim{\mathcal{D}}}\Big[\log\sigma\big(\beta\frac{\pi_{\bm{\theta}}(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\beta\frac{\pi_{\bm{\theta}}(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}\big)\Big]. (3) Accordingly, DPO requires data with contrastive pairs of \{y^{+},y^{-}\} . Later, Negative Preference Optimization (NPO) adopts this preference optimization idea for unlearning, by treating the unlearning sample as undesirable \tau^{-} , and only optimizing the tractable component when \tau^{+} is absent: \displaystyle\min_{\bm{\theta}}{\mathcal{L}}_{\text{NPO}}=-\frac{2}{\beta}\,\mathbb{E}_{\tau^{-}=(x,y)\sim{\mathcal{D}}}\left[\log\sigma\!\left(-\beta\log\frac{\pi_{\bm{\theta}}(y|x)}{\pi_{\text{ref}}(y|x)}\right)\right]. (4) While NPO is designed to be retention-data free, it is often empirically combined with a retention objective e.g. {\mathcal{L}}_{\text{retain}}^{\text{KL}} , requiring retention data and a reference model to avoid catastrophic forgetting on general domain knowledge [ 42 ] . 3 Methods Below we introduce our main idea of effective LLM unlearning, which formulates unlearning as a preference optimization over model policies , in contrast to conventional alignment methods that optimize preference over data samples . 3.1 Negative Preference Alignment As Policy Ranking: Consider a sample trajectory \tau containing an input and response pair \tau=(x,y) , an LLM \pi , and let P(\tau|\pi)=\pi(y|x)\cdot p(x), where p(x) does not depend on \pi , we denote P(\pi|\tau)=\frac{P(\pi).P(\tau|\pi)}{P(\tau)}\propto P(\pi).P(\tau|\pi) to represent the likelihood that the observed response in \tau is generated by \pi . Built on the Bradley-Terry model [ 3 ] , for an arbitrary reference policy \pi_{\beta} , we denote P(\pi_{\bm{\theta}}\succ\pi_{\beta}|\tau) to quantify the probability that the observed \tau is generated by the target policy \pi_{\bm{\theta}} rather than \pi_{\beta} (see Appendix A.2 for details): \displaystyle\small P(\pi_{\bm{\theta}}\succ\pi_{\beta}|\tau)=\frac{\exp(u(\pi_{\bm{\theta}},\tau))}{\exp(u(\pi_{\bm{\theta}},\tau))+\exp(u(\pi_{\beta},\tau))}=\sigma(\beta\log\frac{\pi_{\bm{\theta}}(y|x)}{\pi_{\beta}(y|x)}), (5) where a log-utility function: u(\pi,\tau)=\log\big(P(\pi|\tau)^{\beta}\big) acts as the negative of energy function in Boltzmann distribution [ 7 ] , a constant term \beta is introduced as an inverse of temperature to smooth optimization, and \sigma(\cdot) is the sigmoid function. When \beta=1 , the utility function simplifies to the standard Bradley–Terry form: P(\pi_{\bm{\theta}}\succ\pi_{\beta}|\tau)_{\beta=1}=\frac{P(\pi_{\bm{\theta}}|\tau)}{P(\pi_{\bm{\theta}}|\tau)+P(\pi_{\beta}|\tau)}. Intuitively, P(\pi_{\bm{\theta}}\succ\pi_{\beta}|\tau) quantifies how well the target policy \pi_{\bm{\theta}} can explain given trajectory, compared to the reference policy \pi_{\beta} . This can be viewed as a preference ranking between two policies based on an observed data sample. Formally, given a dataset \mathcal{D} that needs to be unlearned \pi_{\bm{\theta}} , we frame unlearning as a negative alignment of preference over a pair of policies : \displaystyle\min_{\pi_{\bm{\theta}}} \displaystyle\mathbb{E}_{\tau=(x,y)\sim\mathcal{D}}\Big[\log~P(\pi_{\bm{\theta}}\succ\pi_{\beta}|\tau)\Big]. (6) In contrast, for conventional alignment methods such as DPO, the preference is applied to pairs of data samples rather than policies (Equation 2 ). Resultingly, our method provides a principled formulation that can be applied to practical scenarios for LLM unlearning, where undesirable data may not come with explicit contrastive counterparts. 3.2 Using Reverse Policy As a Counterfactual Reference Up to now, a key question is how to choose the reference policy \pi_{\beta} . Prior art mostly adopts the pre-alignment policy model as a static reference, i.e. \pi_{\beta}\equiv\pi_{\bm{\theta}}|_{t=0} , commonly denoted as \pi_{\text{ref}} . One limitation is that such reference in \log\frac{\pi_{\bm{\theta}}(y|x)}{\pi_{\text{ref}}(y|x)} may become constraints as training evolves, especially for regions x,y where \pi_{\text{ref}} put a high density \pi_{\text{ref}}(y|x)>1-\epsilon , thus only a small margin remains to guide the target policy \pi_{\bm{\theta}} during training, and the effect of such training sample diminishes quickly given a static reference model. To address the above limitations, we follow two principles: i) an ideal reference model should be calibrated to reflect the varying importance of different training samples.Thus, data points for which the model is more confident should contribute more to gradient updates and incur greater penalties during unlearning training; ii) The reference \pi_{\beta} should be adaptive along with the target policy \pi_{\bm{\theta}} . In response, we propose an adaptive reference model: {\pi_{\beta}}(\cdot|x)\equiv 1-\pi_{\bm{\theta}}(\cdot|x) , which approximates an un-normalized probability that reverses the choice of \pi_{\bm{\theta}} given arbitrary input x . The relative margin between the target model \pi_{\bm{\theta}}(y|x) and the reference model 1-\pi_{\bm{\theta}}(y|x) naturally reflects the model’s confidence in y given x : Specifically, when \pi_{\bm{\theta}}(y|x)>1-\epsilon , the rescaling factor \frac{1}{1-\pi_{\bm{\theta}}(y|x)}>\frac{1}{\epsilon} becomes large, and vice versa. Accordingly, a sample response y that yields a high \pi_{\bm{\theta}}(y|x) will lead to an amplified penalty of loss, ascribed to our choice of reverse model as a reference. We use \hat{\pi_{\bm{\theta}}} to indicate a gradient-free version ( \text{grad}(\hat{\pi}_{\bm{\theta}})=\texttt{False} ), and derive the following objective: \displaystyle\min_{{\bm{\theta}}}\mathbb{E}_{\tau\sim\mathcal{D}}\Big[\log~P(\pi_{\bm{\theta}}\succ\pi_{\beta}|\tau)\Big]\equiv\min_{\bm{\theta}}\mathbb{E}_{x,y\sim\mathcal{D}}\Big[-\log~\Big(1-\sigma\big(\beta\log\frac{\pi_{\bm{\theta}}({y}|x)}{1-\hat{\pi_{{\bm{\theta}}}}({y}|x)}\big)\Big)\Big]. (7) 3.3 Tokenized Unlearning Optimization Another pain-point for alignment-based methods is the length bias incurred by samples with varying token sizes |y| . In practice, \log\pi_{\bm{\theta}}(y|x)=\sum_{i=1}^{|y|}\log\pi_{\bm{\theta}}({y}_{i}|x,{y}_{0 , which encourages higher \pi_{\bm{\theta}}(\cdot|x) assigned to desirable responses. Similar insights were later applied to an unlearning method dubbed SimNPO [ 13 ] that combines the merits of NPO and SimPO: \min_{\bm{\theta}}{\mathcal{L}}_{\text{simNPO}}\equiv-\frac{2}{\beta}\sigma(-\frac{\beta}{|y|}\log\pi_{\bm{\theta}}(y|x)-\gamma). Contrary to the prior work that involves an extra margin term \gamma , we turn the curse of data length bias into a blessing: we frame each conditional token generation \pi(y_{i}|x,y_{