GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints

Andy Zhu, Rongzhe Wei, Yupu Gu, Pan Li · 2026

Read GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints by Andy Zhu, Rongzhe Wei, Yupu Gu, Pan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16905.txt.

Introduction

Andy Zhu, Rongzhe Wei, Yupu Gu, and Pan Li teach chatbots to drop taught facts on call. Such fact drops, called unlearning, serve safety and law. Past tools work for plain dense models. Mix-of-experts models route each ask to a few parts. Past tools game that route to dodge facts. The facts stay, but are just hid from view.

Their 2026 preprint, GRIP, curbs the router with shape rules. Those rules force true erasure, not mere dodges. Use loss is kept low as facts drop. Safe drops matter for private text in training sets. The tool works across drop plans, by the team's claim.

Catalog record

CREATORAndy Zhu, Rongzhe Wei, Yupu Gu, Pan Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16905

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints Thanks: [email protected] Andy Zhu, Rongzhe Wei, Yupu Gu, Pan Li Abstract Machine unlearning (MU) for large language models has become critical for AI safety, yet existing methods fail to generalize to Mixture-of-Experts (MoE) architectures. We identify that traditional unlearning methods exploit MoE’s architectural vulnerability: they manipulate routers to redirect queries away from knowledgeable experts rather than erasing knowledge, causing a loss of model utility and superficial forgetting. We propose Geometric Routing Invariance Preservation (GRIP), an algorithm-agnostic framework for unlearning for MoE. Our core contribution is a geometric constraint, implemented by projecting router gradient updates into an expert-specific null-space. Crucially, this decouples routing stability from parameter rigidity: while discrete expert selections remain stable for retained knowledge, the continuous router parameters remain plastic within the null space, allowing the model to undergo necessary internal reconfiguration to satisfy unlearning objectives. This forces the unlearning optimization to erase knowledge directly from expert parameters rather than exploiting the superficial router manipulation shortcut. GRIP functions as an adapter, constraining router parameter updates without modifying the underlying unlearning algorithm. Extensive experiments on large-scale MoE models demonstrate that our adapter eliminates expert selection shift (achieving over 95% routing stability) across all tested unlearning methods while preserving their utility. By preventing existing algorithms from exploiting MoE model’s router vulnerability, GRIP adapts existing unlearning research from dense architectures to MoEs. 1 Introduction Large language models have achieved remarkable progress in the understanding and generation of natural language, transforming domains from scientific research to creative writing ( 4 ; 32 ; 14 ) . Despite their success, these models can often unintentionally memorize sensitive information from training data, including personal details, proprietary knowledge, and harmful content ( 6 ; 38 ; 26 ) . Machine unlearning (MU) has emerged as a potential solution to this vulnerability, providing mechanisms to selectively remove information from already trained models ( 3 ; 5 ; 27 ) . This capability is essential not only for regulatory compliance with privacy laws such as GDPR’s “right to be forgotten” ( 11 ; 33 ) , but also for maintaining model safety by removing biased, outdated, or harmful knowledge post-deployment ( 24 ; 34 ) . While retraining from scratch provides the gold standard for machine unlearning, its computational cost has motivated the development of more efficient methods that manipulate model knowledge through targeted optimization ( 17 ; 7 ) . The majority of frontier approaches ( 36 ; 21 ; 15 ; 23 ; 18 ; 20 ; 39 ) share a core strategy of optimizing an unlearning objective (e.g., maximizing loss on forget data) to intentionally degrade performance on target data complemented with various methods to maintain general model capabilities. However, such methods targeting LLMs are primarily deployed and validated on dense architectures, assuming uniform parameter engagement for every input. This assumption is violated by the sparse models increasingly deployed at the frontier. Recently, frontier LLMs have begun to primarily adopt sparse Mixture-of-Experts (MoE) architectures ( 12 ; 10 ; 19 ; 8 ) . In this architecture, each transformer layer contains multiple distinct feed-forward networks, or experts, and a router network that selects a small subset of these experts to process each input token, activating only a fraction of the model’s parameters per input. When applied to these models, existing unlearning methods fail catastrophically through expert selection shift ( 42 ) : rather than erasing knowledge from expert parameters, they manipulate routers to bypass knowledgeable experts. This routing disruption cascades through network depth as early-layer shifts alter representations flowing to later routers, causing severe utility degradation across all queries. Moreover, targeted information remains recoverable in bypassed experts, indicating superficial rather than genuine forgetting. To this end, we propose Geometric Routing Invariance Preservation (GRIP) , an algorithm-agnostic framework that addresses this gap. GRIP introduces hard geometric constraints by projecting router gradient updates into subspaces that preserve routing decisions. Crucially, this formulation decouples routing stability from parameter rigidity; it forces the optimization to erase knowledge from expert parameters while allowing the router to dynamically adapt in directions orthogonal to the expert selecting vector of the retain set, preventing the model from becoming static. To validate this framework, we conduct an extensive empirical evaluation on a 30 billion parameter MoE model, spanning hazardous knowledge removal (WMDP) and copyright protection (MUSE), ensuring broad applicability by testing GRIP across four distinct unlearning paradigms. We further rigorously probe the mechanism of unlearning through layer-wise routing analysis and adversarial expert forcing attacks, designed to determine whether knowledge is genuinely erased or merely hidden by router manipulation. Our results demonstrate that GRIP eliminates the catastrophic trade-off observed in unconstrained baselines. We (i) restore routing stability from 0.21 to >0.94 across all methods, (ii) improve retain accuracy by over 85\% to match dense model baselines, and (iii) achieve genuine knowledge erasure where baselines fail by reducing adversarial knowledge recovery from 61\% to just 3\% . These gains are achieved with minimal computational overhead ( 1.2\times training time) via our Post-Training Correction, establishing geometric constraints as a scalable, algorithm-agnostic solution for safe MoE deployment. Figure 1: Geometric Routing Invariance Preservation (GRIP) prevents expert selection collapse. (a) The Problem: Standard unlearning methods suffer from Expert Selection Shift, where the router acts as an optimization shortcut, diverting queries away from knowledgeable experts rather than erasing information. This causes Routing Stability (RS), a measure of the consistency of expert selection we define formally in Eq ( 2 ), to collapse from 1.0 to \approx 0.2 . (b) Our Solution: GRIP introduces two mechanisms to decouple routing stability from parameter plasticity: (a) Training-Time Enforcement, which projects gradients into a "safe" null-space orthogonal to the router’s decision boundaries; and (b) Post-Training Correction, which analytically realigns a drifted router after unlearning. Both methods restore stability to >0.94 while preserving unlearning efficacy. 2 Related Work Unlearning Methods for LLMs. Machine unlearning (MU) has recently gained significant attention for removing unwanted information from trained models without costly retraining ( 3 ; 27 ) . Existing methods approach this challenge through various optimization strategies. Many formulate unlearning as an objective, such as directly maximizing the loss on forget data via gradient ascent ( 17 ; 7 ) , often regularized by a KL-divergence term against the original model to preserve retain set utility. Others leverage preference learning, such as Negative Preference Optimization (NPO), which trains the model to favor a generic, non-committal output over the specific information designated for removal ( 39 ; 20 ) . Alternatively, some methods intervene at the activation level; for instance, RMU (Representation Misdirection) identifies and nullifies internal representations that encode target knowledge ( 22 ; 44 ) . However, these methods are primarily designed with the assumption of a dense architecture. This assumption is violated by sparse MoEs, as their routers provide an optimization shortcut that these methods exploit. Consequently, the only prior work on MoE unlearning, SEUF ( 42 ) , relies on soft regularization penalties and restricted expert updates, which risks incomplete forgetting. Our framework, in contrast, addresses these limitations with hard geometric constraints that guarantee routing preservation while enabling comprehensive parameter updates. The Rise of Mixture-of-Experts Architectures. Sparse MoE architectures have become ubiquitous in frontier

Read the complete text