Improving LLM Unlearning Robustness via Random Perturbations

Dang Huu-Tien, Hoang Thanh-Tung ‡, Anh Tuan Bui, Phuong Minh Nguyen †, Le-Minh Nguyen †, Naoya Inoue † Japan Advanced Institute of Science and Technology, ‡ VNU University of Engineering and Technology, · 2025

Read Improving LLM Unlearning Robustness via Random Perturbations by Dang Huu-Tien, Hoang Thanh-Tung ‡, Anh Tuan Bui, Phuong Minh Nguyen †, Le-Minh Nguyen †, Naoya Inoue † Japan Advanced Institute of Science and Technology, ‡ VNU University of Engineering and Technology, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19202.txt.

Description

Dang Huu-Tien and co-authors show that teaching a language model to forget can quietly make it fragile, in a 2025 paper. Unlearning methods aim to erase chosen knowledge. The team found that even one harmless forget-related token in a normal query can make an unlearned model misbehave. They set out to explain why.

Their theory recasts unlearning as a backdoor process. Forgetting links the trigger tokens to target outputs, much like an attack would. So the tokens act as hidden triggers, and the knowledge is masked, not erased. To fix the weakness, they reinterpret the training data that should be kept. Random perturbation ideas guide the defense. The work warns that erasure claims need stress tests.

Catalog record

CREATORDang Huu-Tien, Hoang Thanh-Tung ‡, Anh Tuan Bui, Phuong Minh Nguyen †, Le-Minh Nguyen †, Naoya Inoue † Japan Advanced Institute of Science and Technology, ‡ VNU University of Engineering and Technology,
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19202

Browse the full library · About the Tower of Babel