A mean teacher algorithm for unlearning of language models

Yegor Klochkov · 2025

Read A mean teacher algorithm for unlearning of language models by Yegor Klochkov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13388.txt.

Introduction

Making a language model forget selected texts without dulling everything else it knows remains an open problem. Yegor Klochkov borrows a tool from continual learning: the mean teacher algorithm, which moves a teacher model slowly and therefore along a gentle path. He shows this slow walk approximates a form of gradient descent that naturally favors flat, low-damage updates to the model's weights.

That gentle method has a weakness, gradients that fade to nothing, so he pairs it with a new loss, negative log-unlikelihood, designed to avoid the vanishing. Together, the pair improved some measures on the MUSE unlearning benchmarks. The paper's contribution is less a new machine than a translation: a stability trick from one corner of machine learning, carried into the forgetting problem with a fix for its known flaw.

Catalog record

CREATORYegor Klochkov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13388

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A mean teacher algorithm for unlearning of language models Yegor Klochkov Abstract One of the goals of language model unlearning is to reduce memorization of selected text instances while retaining the model’s general abilities. Despite various proposed methods, reducing memorization of large datasets without noticeable degradation in model utility remains challenging. In this paper, we investigate the mean teacher algorithm ( Tarvainen & Valpola, 2017 ) , a simple proximal optimization method from continual learning literature that gradually modifies the teacher model. We show that the mean teacher can approximate a trajectory of a slow natural gradient descent (NGD), which inherently seeks low-curvature updates that are less likely to degrade the model utility. While slow NGD can suffer from vanishing gradients, we introduce a new unlearning loss called “negative log-unlikelihood” (NLUL) that avoids this problem. We show that the combination of mean teacher and NLUL improves some metrics on the MUSE benchmarks ( Shi et al., 2024 ) . 1 Introduction In the recent years, Large Language Models (LLMs) have reached unprecedented capabilities that are achieved through training on massive datasets. Often comprising hundreds of billions of tokens, these datasets are collected from diverse internet sources. This data-intensive approach raises significant ethical concerns as some protected user and copyright data may be compromised. The General Data Protection Regulation (GDPR) establishes fundamental rights for individuals, including the “right to be forgotten,” which grants the deletion of personal data upon request. Concurrently, the inclusion of copyrighted content in training datasets has initiated legal quarrels ( Doe 1 v. GitHub, 2022 ; Tremblay v. OpenAI, 2023 ) . These regulatory and legal actions have sparked interest in methods that would reduce the influence of selective training data on predictions of large pretrained models, a process known as machine unlearning ( Bourtoule et al., 2021 ; Ginart et al., 2019 ) . In unlearning of language models, we are aimed at modifying the model weights that would make them behave as if it was never trained on particular selected data, which we refer to as forget set ( Yao et al., 2023 ; Eldan & Russinovich, 2023 ; Cooper et al., 2024 ; Shi et al., 2024 ) . For example, we want to suppress certain outputs, like being able to reproduce some memorized training instances word-by-word with completion requests, or answering questions about their content. Following ( Shi et al., 2024 ) , we refer to the completion ability as verbatim memorization , and the ability to answer questions as knowledge memorization . We therefore want to reduce such memorization, ideally, without decreasing the model’s utility. There have been a lot of interest in the recent literature towards language model unlearning, with various methods proposed. A generally accepted approach consists of finetuning a model by optimizing an objective specifically designed to discourage memorization. Much of research is dedicated to designing specific loss functions that encourage unlearning, with notable examples such as log-likelihood (referred to as gradient ascent) ( Thudi et al., 2022 ; Yao et al., 2023 ) , Negative Preference Optimization ( Zhang et al., 2024a ) , and others ( Chundawat et al., 2023 ; Fan et al., 2024 ; Wang et al., 2024 ; Li et al., 2024 ) . With the resulting objective including a regularization term, it is typically optimized with AdamW ( Loshchilov et al., 2017 ) , which is a standard way of optimizing language models. A few recent works dedicated to unlearning address the question of how we optimize the corresponding objective, Wang et al. (2025) employ some gradient projections, and Bu et al. (2024) consider adaptive learning rates designed specifically for unlearning. We revisit a simple method called mean teacher ( Tarvainen & Valpola, 2017 ) , which so far appeared only in the context of continual learning literature. We show that in certain scenarios, it can approximate the trajectory of a slow natural gradient descent. This means that the algorithm performs updates along the low curvature directions, which can help retain the model utility and avoid neural collapse. However, we show that mean teacher can suffer from vanishing gradients with some popular choices of unlearning losses, such as log-likelihood and NPO. Instead, we introduce a very simple loss called Negative Log UnLikelihood (NLUL) that avoids this problem and combines well with mean teacher. For the experiments, we focus on the MUSE benchmark ( Shi et al., 2024 ) , which contains rather large forget set (3.3M tokens). Although there are methods that are capable of removing verbatim memorization, the reported results so far show it is challenging to reduce knowledge memorization without dramatic reduction in the utility. We show that one of the versions of mean teacher can achieve this. However, we show that in this case the reduction of knowledge memorization is accompanied by reduction of the Massive Multitask Language Understanding metric (MMLU) ( Hendrycks et al., 2020 ) . This means that the evaluations in the MUSE-benchmark may not be enough for adequate assessment of unlearning algorithms that researchers state reduce knowledge memorization. On the other hand, there are variants that achieve competititve verbatim memorization reduction and utility preservation, while improving on another metrics that are part of the MUSE benchmark associated with risks of privacy leakage. In our work, we focus exclusively on methods that edit the model weights. We mention that some methods are based on expanding the model decision space through guardrails ( Thaker et al., 2024b ; Liu et al., 2024 ) and training low-rank adapters on top of the pretrained model weights ( Gao et al., 2024 ; Ji et al., 2024 ) . Unlearning is sometimes seen as a safety mechanism, with focus on reducing potentially harmful knowledge of language models ( Li et al., 2024 ) . There, the focus is on removing knowledge of whole topics or concepts. On the other hand, we focus on unlearning independent text instances. 2 Unlearning set-up and description of the method Most unlearning benchmarks assume that we are given an access to two datasets: • forget set D_{f} is a set of training instances that we know are the source of knowledge we want to remove; • retain set D_{r} is another set of training instances that do not contain undesirable knowledge, which is intended to assist in retaining the model’s utility. We acknowledge knowing the source of undesirable knowledge would already be a strong assumption. However, most of the existing benchmarks follow such set up ( Maini et al., 2024 ; Shi et al., 2024 ) , see also a review in ( Thaker et al., 2024a ) . Often, unlearning is performed by optimizing an objective of the form, L_{forget}(D_{f};\theta)+L_{retain}(D_{r};\theta), (1) where L_{forget} averages some unlearning loss on the given examples. That is, L_{forget}(D_{f};\theta)=\frac{1}{|D_{f}|}\sum_{(x,y)\in D_{f}}\ell(h(x;\theta),y), (2) where h(x;\theta) are the logit outputs of the model on the given instance x , and y is the target label. One popular choice of unlearning loss, is the log likelihood (LL) \ell_{LL}(h,y)=\log\mathrm{sf}(h)_{y}, where \mathrm{sf}(h)_{y}=e^{h_{y}}/\left(\sum_{j}e^{h_{j}}\right) is the softmax function. Such loss is a straightforward negation of the standard learning loss—negative log likelihood (NLL)—and it is intended to discourage correct prediction on the inputs from the forget set. Other unlearning

Read the complete text