UCD: Unlearning in LLMs via Contrastive Decoding

Vinith Suriyakumar MIT, Ayush Sekhari Boston University Ashia Wilson MIT · 2025

Read UCD: Unlearning in LLMs via Contrastive Decoding by Vinith Suriyakumar MIT, Ayush Sekhari Boston University Ashia Wilson MIT (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12097.txt.

Introduction

Suriyakumar, Sekhari, and Wilson set out UCD, a run-time way to un-teach chat tools. Un-teach aims to strip set facts from a tool but keep its skill. Facts in view may be soft or banned for the tool's use. Their way steers outs by twin small tools at run time. One small tool trained sans the strip set, one with it in. The gap twixt those two guides the big tool's outs.

That steer lifts the trade twixt strip force and kept skill. Two test sets, TOFU and MUSE, grade the way in runs. Gains show in strip grade and kept scores next to past ways. The team reads twin steer as a cheap, apt path here. It suits vast tools where re-train costs run too high.

Catalog record

CREATORVinith Suriyakumar MIT, Ayush Sekhari Boston University Ashia Wilson MIT
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12097

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UCD: Unlearning in LLMs via Contrastive Decoding Vinith Suriyakumar MIT, Ayush Sekhari Boston University Ashia Wilson MIT Abstract Machine unlearning aims to remove specific information, e.g. sensitive or undesirable content, from large language models (LLMs) while preserving overall performance. We propose an inference-time unlearning algorithm that uses contrastive decoding, leveraging two auxiliary smaller models, one trained without the forget set and one trained with it, to guide the outputs of the original model using their difference during inference. Our strategy substantially improves the tradeoff between unlearning effectiveness and model utility. We evaluate our approach on two unlearning benchmarks, TOFU and MUSE. Results show notable gains in both forget quality and retained performance in comparison to prior approaches, suggesting that incorporating contrastive decoding can offer an efficient, practical avenue for unlearning concepts in large-scale models. 1 Introduction Large Language Models (LLMs) achieve impressive general capabilities thanks to massive training datasets and compute. However, these capabilities raise significant safety and security concerns, including copyright violations ( Karamolegkou et al. 2023 ) , harmful content generation, and retention of dangerous knowledge (e.g., bioweapon instructions) ( Shevlane et al. 2023 ) . Retraining models to address these issues by excluding problematic data is impractical at scale. This has led to growing interest in efficient methods for machine unlearning , which aim to remove specific information from trained models without retraining. The field of machine unlearning began with a focus on removing the influence of specific training data points from trained machine learning models ( Cao & Yang 2015 ; Bourtoule et al. 2021 ; Neel et al. 2021 ; Sekhari et al. 2021 ; Ghazi et al. 2023 ; Suriyakumar & Wilson 2022 ) . This initial motivation arose primarily from compliance with emerging privacy regulations, such as the EU’s General Data Protection Regulation (GDPR) ( Voigt & Von dem Bussche 2017 ) and the California Consumer Privacy Act (CCPA) ( CCP 2018 ) , both of which enforce the Right to be Forgotten . More recently, researchers concerned with AI safety have broadened the scope of machine unlearning to also include removing unwanted or harmful knowledge from large language models ( Li et al. 2024 ; Barez et al. 2025 ; Zhang et al. 2023b ) . So far, two broad classes of unlearning algorithms have been proposed for LLM unlearning: finetuning-based approaches and representation-engineering approaches. Finetuning algorithms define an objective to represent “unlearning” and optimize it using samples of data to be forgotten (i.e., the forget set ). The canonical example is gradient ascent, which maximizes empirical loss on the forget set. Extensions of this approach incorporate additional loss terms to maintain model utility ( Jang et al. 2022 ; Yao et al. 2023a ; Chen & Yang 2023 ; Schwarzschild et al. 2024 ) or modify alignment procedures, such as direct preference optimization ( Rafailov et al. 2024 ) . Representation-engineering methods propose objectives to modify internal representations of the model with respect to the forget set, typically by projecting them onto random or orthogonal subspaces ( Li et al. 2024 ) . A shortcoming of both of these classes of methods is that they are expensive to run and suffer from poor forget-utility tradeoffs ( Shi et al. 2024 ) . Motivated by recent advances in inference-time methods that improve reasoning and alignment without extensive retraining, we propose Unlearning via Contrastive Decoding (UCD) , a novel inference-time unlearning algorithm inspired by contrastive decoding principles ( Li et al. 2023 ) . UCD leverages two small auxiliary models, one trained exclusively on the forget set and another trained on the retain set, allowing it to effectively remove undesirable knowledge at inference (Figure 1 ). This approach significantly improves the forget-utility tradeoff and sets new state-of-the-art benchmarks on established unlearning datasets (TOFU and MUSE News). Additionally, due to its computational efficiency, UCD enables practical unlearning even on extremely large models such as Llama2-70B, a task previously infeasible with existing approaches. Our main contributions are: • Contrastive Decoding-Based Unlearning: We introduce UCD, an efficient inference-time unlearning algorithm utilizing two auxiliary models trained separately on forget and retain data. Whenever it is possible to obtain a clean model trained solely on the forget set, or when the data is sufficiently separable to allow targeted fine-tuning, UCD can be easily applied. • Superior Forget-Utility Tradeoff: UCD significantly outperforms existing methods on standard machine unlearning benchmarks (TOFU, MUSE), achieving forget performance indistinguishable from retraining and improved utility due to contrastive decoding’s enhanced text quality. • Scalability to Significantly Larger Models: Unlike existing weight-modifying unlearning methods constrained by computational costs, UCD demonstrates practical inference-time unlearning on significantly larger models, including Llama2-13B and Llama2-70B, only requiring 2 L40s for unlearning on Llama2-13B and 4 NVIDIA H200s for unlearning on Llama2-70B. Whereas all pre-existing baselines required at least 2 A100s for unlearning on Llama2-13B and are infeasible on Llama2-70B on 8 H200s. Figure 1: Illustration of contrastive decoding at inference time in UCD. In the top row, we prompt the model with a sentence from our forget set corpus. The first distribution represents the original model we would like to unlearn from. The difference between our two auxiliary models guides the distribution to suppress the information related to Harry Potter. Meanwhile, in the second row, on a prompt we would like to retain the difference remains small leaving the output unaffected. 2 Background and Related Work Machine unlearning. Our work builds on a growing body of research on machine unlearning ( Bourtoule et al. 2021 ; Nguyen et al. 2022 ; Cao & Yang 2015 ; Gupta et al. 2021 ; Suriyakumar & Wilson 2022 ; Sekhari et al. 2021 ; Ghazi et al. 2023 ; Kurmanji et al. 2023 ; Lev & Wilson 2024 ; Łucki et al. 2024 ) , which aims to develop methods that efficiently modify trained machine learning models to forget specific portions of their training data. In the case of classical discriminative models, the motivation often stems from privacy concerns, particularly the need to protect individuals whose data may have been used during training. A major driver behind this line of research was the introduction of Article 17 of the European Union’s General Data Protection Regulation (GDPR), which codifies an individual’s “right to be forgotten” ( European Union 2016 ) . Various other legislations have followed including California Consumer Privacy Act (CCPA), Canada’s proposed Consumer Privacy Protection Act (CPPA)), and more recently in Australia ( Karp 2023 ) . More recently, the scope of machine unlearning has expanded to include modern generative AI models, which pose additional challenges such as the potential reproduction of copyrighted material, generation of harmful or explicit content, and leakage of sensitive training data ( Zhang et al. 2023a ; Carlini et al. 2021 ) . Unlearning and alignment in LLMs. Machine unlearning for Large Language Models (LLMs) has emerged as a rapidly growing area of research ( Liu et al. 2024a ; Jang et al. 2022 ; Kumar et al. 2022 ; Zhang et al. 2023b ; Pawelczyk et al. 2023 ; Yao et al. 2023a ; Zhang et al. 2024 ; Wang et al. 2024 ; Jia et al. 2024 ; Lu et al. 2022 ; Liu

Read the complete text