EvoMU: Evolutionary Machine Unlearning

Paweł Batorski, Paul Swoboda · 2026

Read EvoMU: Evolutionary Machine Unlearning by Paweł Batorski, Paul Swoboda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02139.txt.

Introduction

Paweł Batorski and Paul Swoboda hunt for loss rules that make a model drop chosen training data while keeping its broad skills. No one rule fits all cases, since how drop and keep sets overlap shifts what works, and candidate rules are vast. Their method, EvoMU, searches that space and breeds task fit rules on its own, with no human in the loop. It beats prior loss based methods on four test sets with a small four billion parameter model.

Catalog record

CREATORPaweł Batorski, Paul Swoboda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86853
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.02139

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EvoMU: Evolutionary Machine Unlearning Paweł Batorski, Paul Swoboda Abstract Machine unlearning aims to unlearn specified training data (e.g. sensitive or copyrighted material). A prominent approach is to fine-tune an existing model with an unlearning loss that retains overall utility. The space of suitable unlearning loss functions is vast, making the search for an optimal loss function daunting. Additionally, there might not even exist a universally optimal loss function: differences in the structure and overlap of the forget and retain data can cause a loss to work well in one setting but over-unlearn or under-unlearn in another. Our approach EvoMU tackles these two challenges simultaneously. An evolutionary search procedure automatically finds task-specific losses in the vast space of possible unlearning loss functions. This allows us to find dataset-specific losses that match or outperform existing losses from the literature, without the need for a human-in-the-loop. This work is therefore an instance of automatic scientific discovery, a.k.a. an AI co-scientist. In contrast to previous AI co-scientist works, we do so on a budget: We achieve SotA results using a small 4B parameter model (Qwen3-4B-Thinking), showing the potential of AI co-scientists with limited computational resources. Our experimental evaluation shows that we surpass previous loss-based unlearning formulations on TOFU- 5 % 5\% , TOFU- 10 % 10\% , MUSE and WMDP by synthesizing novel unlearning losses. Our code is available at https://github.com/Batorskq/EvoMU Figure 1 : High level overview of EvoMU. A LLM proposes candidate unlearning loss functions along with their training budget. For each candidate, we fine-tune the base model with LoRA adapters on forget and retain data, then evaluate the resulting checkpoint using standard forgetting and utility metrics. We select the top- K objectives and summarize their full feedback (loss code, training curves, and evaluation scores). Conditioned on this feedback, the LLM mutates the objectives by adjusting coefficients, modifying structure, and optionally changing the training budget, to produce the next generation. Repeating this verify–select–mutate loop yields task-specific unlearning losses that match or outperform human-designed ones. 1 Introduction As large language models (LLMs) grow in capability, they also increasingly memorize and reproduce sensitive, harmful, or copyrighted content [ Huang et al. 2024b , Carlini et al. 2022 , Wang et al. 2023a , Wei et al. 2024a , Ousidhoum et al. 2021 , Barbulescu and Triantafillou 2024 , Dou et al. 2025a ] . In many real deployments, this creates a practical need for machine unlearning : removing the influence of specified training data from a trained model without retraining from scratch [ Cao and Yang 2015 , Ginart et al. 2019 , Bourtoule et al. 2021 ] . Beyond its origins in data deletion and privacy regulations (e.g., the “right to be forgotten” [ Rosen 2011 , Hoofnagle et al. 2019 ] ), unlearning is now motivated by content compliance, safety, and intellectual property concerns [ Liu et al. 2025a ] . A dominant approach for unlearning in modern LLMs is post-training fine-tuning with unlearning losses that suppress unlearning data while preserving general utility. They include gradient-ascent or gradient-difference formulations [ Yao et al. 2024b , Liu et al. 2022 , Maini et al. 2024b ] , preference-optimization objectives that discourage undesired completions [ Zhang et al. 2024a , Fan et al. 2024a , Mekala et al. 2025a ] , and distillation or logit/representation-regularization schemes [ Wang et al. 2024a , Dong et al. 2025a , Li et al. 2024 ] . However, loss design remains a major bottleneck. Hand-designed losses may be brittle: small changes in coefficients, margins, or reference signals can shift the trade-off between effective forgetting and over-unlearning. Additionally, unlearning losses are rarely universal. The difficulty of selective forgetting depends strongly on the structure of the unlearning instance: how concentrated or diffuse the target content is, how much the forget and retain distributions overlap, and whether the forget targets are isolated facts, stylistic patterns, or long-form passages. This structure changes the dominant failure mode. When forget and retain share entities, phrasing, or underlying facts, a naive loss may suppress too broad regions of behavior (e.g., refusing or becoming uncertain in-domain) rather than selectively removing the intended information. What matters is not only “how much forgetting” is applied, but how the loss balances forget vs. retain and how it prevents degenerate solutions. These considerations suggest that loss design should adapt to the data geometry of the unlearning problem rather than rely on a single fixed loss. The above observation, namely large and problem-dependant, yet well-specified search space with clear evaluation protocol, makes machine unlearning a good target for automated scientific discovery (ASD). Modern ASD has seen great success (i) when using LLMs for code generation, experimentation and iterative refinement [ Wang et al. 2023b , Lu et al. 2024a , Schmidgall et al. 2025a , Jansen et al. 2025a , Yamada et al. 2025a ] and (ii) when candidates are executable and progress is measured by automatic metrics, enable efficient, auditable search loops (e.g., algorithm or reward-function discovery) [ Fawzi et al. 2022 , Ma et al. 2023 , Richardson et al. 2024a , Cheng et al. 2025b ] . In this work, we study unlearning loss design as an ASD problem and introduce EvoMU, an evolutionary objective-discovery framework. EvoMU uses a code-generating LLM to propose candidate unlearning loss, fine-tunes lightweight LoRA adapters for each candidate, evaluates with standardized unlearning metrics, and iteratively refines the best loss using an evolutionary search. Crucially, we show that this discovery loop does not require frontier-scale or closed-weight models: a small open model (Qwen3-4B-Thinking) is sufficient to synthesize novel, effective losses that match or outperform strong human-designed baselines across multiple benchmarks. Our contributions are: • ASD for unlearning. We formulate unlearning loss design as an automated discovery problem and present an end-to-end pipeline that proposes, trains, evaluates, and refines executable loss functions. • Small-model effectiveness. We demonstrate that a 4B-parameter open LLM can effectively drive loss discovery, contrasting with prior ASD systems that rely on very large or closed models. • Task-specific losses matter. Across TOFU, MUSE, and WMDP, the best-performing losses differ substantially, indicating that unlearning losses should be tailored to the forget/retain data structure rather than treated as universal. • LLM-sampled losses are a strong baseline. Even without refinement, randomly sampled candidate losses can be competitive with established losses (e.g., NPO), highlighting that loss choice is a major lever. • Empirical structure of effective losses. Across benchmarks, many top-performing discovered losses are simple and often do not require explicit reference-model terms, suggesting underexplored regions of the unlearning loss space. 2 Related Work 2.1 Machine Unlearning for LLMs. Recent surveys and position papers argue that unlearning for LLMs should be viewed as a continuum of intervention surfaces, spanning inference-time suppression, parameter editing, and weight-level updates, while emphasizing that reliable evaluation and meaningful guarantees remain open problems [ Liu et al. 2025a , Ren et al. 2025a , Thaker et al. 2024a ] . Broadly, existing approaches fall into three (overlapping) families: (i) mitigation mechanisms that restrict access to unwanted content without directly optimizing a forgetting loss, (ii) localized editing or model-merging methods that overwrite specific associations, and (iii) training-time unlearning that updates model parameters (often via parameter-efficient fine-tuning) under a designed loss. Mitigation without optimizing

Read the complete text