LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks

Soumyadeep Pal Changsheng Wang James Diffenderfer, Bhavya Kailkhura Sijia Liu · 2025

Read LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks by Soumyadeep Pal Changsheng Wang James Diffenderfer, Bhavya Kailkhura Sijia Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10185.txt.

Description

Soumyadeep Pal and co-authors report a surprising finding about tests used to judge whether a language model has forgotten unwanted data, in a paper from 2025. Unlearning aims to remove the effect of bad training data while keeping the model useful. Two popular benchmarks are widely used to grade unlearning methods.

The team found that unlearning with the full forget set can be matched by a tiny random subset, as small as five percent of it. The effect held across methods and across ways of picking the subset. They trace it to a compact set of high impact keywords in the forget data. The result suggests current benchmarks make unlearning look easier than it may be in general.

Catalog record

CREATORSoumyadeep Pal Changsheng Wang James Diffenderfer, Bhavya Kailkhura Sijia Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10185

Browse the full library · About the Tower of Babel