Auditing Language Model Unlearning via Information Decomposition
Anmol Goel, Alan Ritter, Iryna Gurevych · 2026
Read Auditing Language Model Unlearning via Information Decomposition by Anmol Goel, Alan Ritter, Iryna Gurevych (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15111.txt.
Introduction
Anmol Goel, Alan Ritter, and Iryna Gurevych audit whether language models truly forget. Their 2026 arXiv paper deals with machine unlearning, methods that claim to erase chosen data from a model. The methods look like they work, yet a problem hides inside.
The authors show that facts about the erased data can still be read off. The traces sit in the model's internal states and can be decoded in a straight-line way. They build an audit frame based on information theory. It uses partial information decomposition to measure what remains after unlearning.
Catalog record
| CREATOR | Anmol Goel, Alan Ritter, Iryna Gurevych |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Auditing Language Model Unlearning via Information Decomposition Anmol Goel, Alan Ritter, Iryna Gurevych Abstract We expose a critical limitation in current approaches to machine unlearning in language models: despite the apparent success of unlearning algorithms, information about the forgotten data remains linearly decodable from internal representations. To systematically assess this discrepancy, we introduce an interpretable, information-theoretic framework for auditing unlearning using Partial Information Decomposition (PID). By comparing model representations before and after unlearning, we decompose the mutual information with the forgotten data into distinct components, formalizing the notions of unlearned and residual knowledge. Our analysis reveals that redundant information, shared across both models, constitutes residual knowledge that persists post-unlearning and correlates with susceptibility to known adversarial reconstruction attacks. Leveraging these insights, we propose a representation-based risk score that can guide abstention on sensitive inputs at inference time, providing a practical mechanism to mitigate privacy leakage. Our work introduces a principled, representation-level audit for unlearning, offering theoretical insight and actionable tools for safer deployment of language models. 1 1 1 Code: https://github.com/UKPLab/eacl2026-auditing-unlearning 1 Introduction The rapid adoption of Large Language Models (LLMs) introduces pressing challenges around user data, including privacy ( 9 ) and copyright concerns ( 30 ) . Studies have shown that LLMs often memorize sensitive information like personally identifiable data ( 42 ; 61 ; 64 ; 44 ; 43 ; 17 ) . Although users may initially consent to data usage, they may later revoke that consent. Legal frameworks like the GDPR 2 2 2 https://gdpr-info.eu/ and the EU AI Act 3 3 3 https://artificialintelligenceact.eu/ enforce the “right to be forgotten,” mandating the ability to remove user data from machine learning systems. This has led to growing interest in machine unlearning methods that aim to eliminate the influence of specific data samples from trained models ( 4 ) . Figure 1: Existing unlearning methods may appear effective but retain forgotten data internally (A). Our framework (B) uses Partial Information Decomposition to reveal residual knowledge, which (C) correlates with attack risk and informs a privacy-aware abstention score. Existing unlearning techniques can be broadly taxonomized as exact or approximate ( 59 ; 6 ) . Exact unlearning involves retraining the model from scratch without the specified data, which is infeasible for large-scale LLMs. Approximate unlearning modifies the model parameters post hoc to minimize the influence of the targeted data. While the latter is more practical, it lacks formal guarantees and often depends on heuristic optimization. Although several approximate unlearning algorithms have emerged, most focus solely on parameter updates without auditing and verifying whether unlearning has actually occurred during the unlearning process ( 52 ) . In practice, empirical auditing is crucial to assess the effectiveness of such algorithms ( 57 ; 54 ) . Current evaluation methods for data removal generally check the presence of forgotten data in the output distributions of LLMs and quantify data leakage in a metric like Forget Quality ( 41 ; 47 ) . Such evaluations suffer from surface-level assessments where a model can learn to alter its output yet still retain internal representations of it. Another line of evaluations include membership inference attacks (MIAs) that employ heuristics to determine the membership of the forgotten data after unlearning (an unsuccessful MIA indicates successful data removal) ( 33 ) . However, recent evidence suggests that MIAs do not work well in LLMs and often perform poorly, even near random guess performance in some cases ( 13 ) . This undermines the objective of an unlearning audit and verification. More broadly, these approaches also fail to quantify how much information was actually removed or retained. Current audits of LLM unlearning are superficial. They typically confirm only weak unlearning ( 26 ) which checks for changes in model outputs while failing to verify the complete removal of information from internal representations. This leaves open the risk of "shallow unlearning," where models create a false sense of privacy while retaining sensitive data, falling short of the strong unlearning mandated by regulations like GDPR ( 7 ) . We argue for a paradigm shift from behavioral checks to principled, information-theoretic auditing. By comparing a model’s representations before and after an unlearning request, we can move beyond if a model’s behavior changed to how much information was truly erased. To achieve this, we introduce a novel framework based on Partial Information Decomposition (PID) ( 40 ) , a powerful tool for dissecting shared information ( 11 ; 23 ; 36 ; 14 ; 10 ) . Our PID-based audit formalizes two key quantities: Unlearned knowledge as the unique information about forgotten data successfully erased from the model and, residual knowledge as the redundant information that persists, shared between the original and unlearned models, representing a potential privacy vulnerability. This work is the first to operationalize PID for unlearning audits. Our contributions are: 1. We provide empirical evidence of shallow unlearning , showing that significant residual knowledge persists even when standard metrics suggest success ( Section 3 ). 2. We provide a formal framework to quantify unlearned and residual knowledge using the information-theoretic concepts of uniqueness and redundancy ( Section 5 ). 3. We demonstrate the practical utility of our audit , showing Residual Knowledge correlates strongly with adversarial vulnerability and can power a privacy-enhancing abstention mechanism ( Section 6.2 ). Ultimately, we offer a novel, more rigorous lens for evaluating LLM unlearning, providing developers and regulators with actionable tools to ensure data is truly forgotten. 2 Related work LLM Unlearning Most existing unlearning methods for LLMs follow traditional machine unlearning approaches to minimize the influence of forget samples via gradient updates during finetuning . The most straightforward approach employs a mixture of forgetting and retaining objectives by performing gradient ascent updates on the non-desirable sequences and regular gradient descent updates on the desirable sequences ( 29 ; 15 ; 25 ) . A variant of direct finetuning is the use of Direct Preference Optimization to unlearn by framing the forget set as a preference set ( 41 ; 63 ) . Another line of unlearning methods involve localized parameter modification instead of full finetuning ( 24 ; 34 ; 58 ; 28 ) . Recent works use auxiliary models for unlearning by training smaller models that can help in modifying the model’s responses such that the forget set responses are not generated ( 27 ; 56 ; 12 ) . A final line of work based on input/output unlearning uses prompt engineering and post-processing methods to alter model behavior on the forget set ( 38 ; 53 ; 45 ) . Since auxiliary and input/output unlearning methods do not modify the model representations/weights, they only work at the behavior level which is often not enough to comply with regulatory frameworks like GDPR ( 16 ) . In this work, we focus on representative, state of the art algorithms from both paradigms of LLM unlearning where model internals are modified - finetuning and localized editing. Unlearning Evaluations Most work on LLM unlearning has primarily relied on output-based or task-centric evaluation metrics ( 52 ) . These include forgetting accuracy ( 39 ) , which measures performance drop on erased data, and membership inference attacks (MIAs) to assess residual memorization ( 48