Leak@ k k : Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

Hadi Reisizadeh, Jiajun Ruan, Yiwei Chen, Soumyadeep Pal, Sijia Liu, Mingyi Hong · 2025

Read Leak@ k k : Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding by Hadi Reisizadeh, Jiajun Ruan, Yiwei Chen, Soumyadeep Pal, Sijia Liu, Mingyi Hong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04934.txt.

Introduction

Hadi Reisizadeh and co-authors test whether methods that make large language models forget still work when text is sampled in the normal random way, not just greedy choice. They study removal of private, toxic, or banned content that rules and ethics require.

The team adds a new score, leak at k, which checks how often removed facts return across k samples. On three known test sets, they find that nearly all current methods leak. Facts that look gone under greedy checks come back under normal sampling, so past tests gave a false sense of safety.

Catalog record

CREATORHadi Reisizadeh, Jiajun Ruan, Yiwei Chen, Soumyadeep Pal, Sijia Liu, Mingyi Hong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04934

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leak@ k k : Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding Hadi Reisizadeh, Jiajun Ruan, Yiwei Chen, Soumyadeep Pal, Sijia Liu, Mingyi Hong Abstract Unlearning in large language models (LLMs) is critical for regulatory compliance and for building ethical generative AI systems that avoid producing private, toxic, illegal, or copyrighted content. Despite rapid progress, in this work we show that almost all existing unlearning methods fail to achieve true forgetting in practice. Specifically, while evaluations of these ‘unlearned’ models under deterministic (greedy) decoding often suggest successful knowledge removal using standard benchmarks (as has been done in the literature), we show that sensitive information reliably resurfaces when models are sampled with standard probabilistic decoding. To rigorously capture this vulnerability, we introduce leak@ k k , a new meta-evaluation metric that quantifies the likelihood of forgotten knowledge reappearing when generating k k samples from the model under realistic decoding strategies. Using three widely adopted benchmarks, TOFU, MUSE, and WMDP, we conduct the first large-scale, systematic study of unlearning reliability using our newly defined leak@ k k metric. Our findings demonstrate that knowledge leakage persists across methods and tasks, underscoring that current state-of-the-art unlearning techniques provide only limited forgetting and highlighting the urgent need for more robust approaches to LLM unlearning. † † footnotetext: \ast Equal contribution. 1 Introduction Large language models (LLMs) have demonstrated an extraordinary ability to generate human-like text 40 . These models are typically pre-trained and fine-tuned on massive datasets collected from the web. However, such datasets often contain harmful, toxic, private, or copyrighted content. This raises significant privacy and ethical concerns, as LLMs may produce biased 19 ; 27 , toxic, private, or illegal responses 28 ; 42 ; 18 ; 38 , and even provide dangerous guidance on developing bioweapons or conducting cyberattacks 3 ; 21 . To address these risks, LLM unlearning has emerged as a promising approach: the goal is to remove undesired knowledge and its downstream effects while preserving overall model utility. Unlearning Algorithms. A growing body of work has proposed different unlearning algorithms, often formulating the task as a trade-off between forgetting targeted information and retaining useful capabilities. Examples include gradient ascent methods 26 , negative preference optimization (NPO) 47 , simplified NPO variants (SimNPO) 12 , representation misdirection (RMU) 21 , and bi-level or multi-task optimization approaches 34 ; 4 . These methods achieve partial success in mitigating unwanted information while preserving model utility. Most approaches rely on supervised unlearning (SU) with token-level cross-entropy loss (see Appendix C ), where the model is trained to assign minimum probability to the reference token at each step. This training strategy enforces behavior aligned with the reference outputs. Conversely, RMU 21 follows an unsupervised strategy where instead of using labeled reference tokens, it modifies hidden representations to shift the model away from the forget set while aiming to maintain performance on the retain set. Benchmarks for Unlearning. Alongside these algorithmic advances, several benchmarks have been introduced to evaluate unlearning performance, such as TOFU 26 , MUSE 36 , WMDP 21 , and the multi-task benchmark LUME 32 . These benchmarks test whether models avoid reproducing sensitive knowledge while continuing to generate accurate and useful outputs on non-forget tasks (see Appendix A for details). A critical limitation, however, is that evaluation in these benchmarks is conducted almost exclusively under deterministic decoding , most often greedy decoding, T\!=\!0 , p\!=\!0 where T is the decoding temperature, and p is the top- p value. In this setting, the model always selects the most probable token at each step. While simple and reproducible, greedy decoding masks the probabilistic nature of LLMs, models may still allocate non-trivial probability mass to sensitive tokens, which remains undetected unless probabilistic decoding (e.g., sampling with T>0 or top- p ) is applied. As a result, benchmarks relying solely on greedy decoding fails to expose residual leakage present in the full output distribution. Challenges. A fundamental limitation in the current unlearning literature lies in the mismatch between evaluation and deployment settings. Nearly all existing benchmarks rely on deterministic decoding, most commonly greedy decoding, where the model always selects the single most probable token at each step. While this setup is convenient for standardized evaluation, it poorly reflects deployed systems, where probabilistic decoding strategies such as temperature sampling or top- p sampling are widely adopted, especially in domains such as conversational agent 16 ; 7 and code generation 6 ; 2 . This reliance on greedy decoding creates a serious blind spot. Greedy outputs are low-diversity and repetitive, and often underestimate the model’s latent knowledge. In contrast, probabilistic decoding exhibits creativity 29 and generates human-preferred text 16 , but it also raises the risk that suppressed knowledge resurfaces if undesired knowledge is not truly forgotten 20 ; 35 . In the unlearning tasks, even a single leaked generation can be catastrophic, for example exposing copyrighted passages or private information. Question Ground Truth According to the excerpt, what is the new bus route number that will replace route 666 to Hel in Poland? 669 Method Greedy Decoding Worst of 64 Samples Original …MZK Gdynia is using the new number 669 to replace its route 666.. 669 Having spent six months of intense work… Retrain 92288B3EU (Northern Ireland); B49DCEB7D8 (Scotland); CBAFCEFBE7 (England) The new route number for the bus to Poland will be 669… GradDiff The BBC is not responsible for the content of external sites. …the route will be replaced by the 669 the excerpt says. NPO Tod The first new debris has been found around the Sh-1… 669 It would seem that someone at the national newspaper BLUR-NPO Glory be to God, our creator, our Lord, our Father, Almighty,… It will be number 669… Table 1 : Examples of generated text from different unlearning methods on the MUSE-News dataset, comparing outputs from the deterministic decoder with the worst-case response among 64 generations from the probabilistic decoder with T=0.2 and p=1.0 . Failed unlearning is indicated by undesired answers in red , successful unlearning by green . For example, as Table 1 illustrates, models that appear to have forgotten sensitive passages under greedy decoding readily regenerate them verbatim once sampled multiple times under the probabilistic decoding. In particular, it presents an example from MUSE-News benchmark using the LLaMA2-7B model. Several unlearning methods, including GradDiff 25 , NPO 47 , and BLUR-NPO 34 are evaluated under two decoding strategies: (1) a greedy decoder, as typically used in current benchmarks, and (2) a probabilistic decoder, where we report the most privacy-leaking response among 64 generations, setting T\!=\!0.2 and p\!=\!1.0 . Hence, existing benchmark results on unlearned models may misleadingly determine that forgetting has succeeded, when in fact this “success” is brittle and fails to generalize to realistic usage. Most unlearning evaluations adopt greedy decoding due to its deterministic and reproducible results, while only a few consider probabilistic decoding. Notably, 35 and 46 first explore probabilistic evaluation. However, their approaches remain limited: (1) they measure only statistical uncertainty in token distributions, without assessing whether the generated outputs convey the semantic content of the forgotten knowledge; (2) they rely on single-generation evaluation, which is problematic because one sample provides only a narrow view of the model’s output space and

Read the complete text