Label Smoothing Improves Gradient Ascent in LLM Unlearning
Zirui Pang Hao Zheng Zhijie Deng Ling Li Zixin Zhong Jiaheng Wei · 2025
Read Label Smoothing Improves Gradient Ascent in LLM Unlearning by Zirui Pang Hao Zheng Zhijie Deng Ling Li Zixin Zhong Jiaheng Wei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22376.txt.
Description
Zirui Pang and five co-authors work on making large language models forget harmful or unwanted knowledge while keeping their general skill. The simplest method, gradient ascent on the data to forget, is unstable and can badly damage the model. Their smoothed version mixes the forget data with several sets of normal data at a chosen rate.
Theory guides how to pick that smoothing rate, balancing forgetting against keeping useful skill. Tests on three benchmarks, covering fake author facts, a book series, and news, show the smoothed method beats plain gradient ascent on all measures. It also ranks near the top among rival methods.
Catalog record
| CREATOR | Zirui Pang Hao Zheng Zhijie Deng Ling Li Zixin Zhong Jiaheng Wei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22376