Label Smoothing Improves Gradient Ascent in LLM Unlearning

Zirui Pang Hao Zheng Zhijie Deng Ling Li Zixin Zhong Jiaheng Wei · 2025

Read Label Smoothing Improves Gradient Ascent in LLM Unlearning by Zirui Pang Hao Zheng Zhijie Deng Ling Li Zixin Zhong Jiaheng Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22376.txt.

Description

Zirui Pang and five co-authors work on making large language models forget harmful or unwanted knowledge while keeping their general skill. The simplest method, gradient ascent on the data to forget, is unstable and can badly damage the model. Their smoothed version mixes the forget data with several sets of normal data at a chosen rate.

Theory guides how to pick that smoothing rate, balancing forgetting against keeping useful skill. Tests on three benchmarks, covering fake author facts, a book series, and news, show the smoothed method beats plain gradient ascent on all measures. It also ranks near the top among rival methods.

Catalog record

CREATORZirui Pang Hao Zheng Zhijie Deng Ling Li Zixin Zhong Jiaheng Wei
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22376

Browse the full library · About the Tower of Babel