From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning

Evan Ning, Wei Xue, Dong Lou, Yike Guo · 2026

Read From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning by Evan Ning, Wei Xue, Dong Lou, Yike Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26629.txt.

Introduction

Evan Ning, Wei Xue, Dong Lou, and Yike Guo fight catastrophic forgetting in language models differently. Continual learning teaches a model new tasks without erasing old ones. Standard methods protect important weights, the model's internal settings. On large language models those methods disappoint. The authors blame polysemanticity: single units mix many meanings, so weight importance cannot isolate knowledge worth protecting.

Their method protects activations instead, the signals flowing through the model. A sparse autoencoder supplies a dictionary of single-meaning features. A compact mask of important features is built from current-task data, and only the mask is kept. Training then balances stability against flexibility in feature space. The authors show the classic weight method is a special case of their formulation. On two continual learning benchmarks, their method achieved the strongest results reported in the study.

Catalog record

CREATOREvan Ning, Wei Xue, Dong Lou, Yike Guo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26629

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning Evan Ning, Wei Xue, Dong Lou, Yike Guo Abstract Weight-space regularization methods such as Elastic Weight Consolidation (EWC) are the standard approach to catastrophic forgetting in continual learning. However, those methods tend to underperform when applied to large language models. We argue that such underperformance can be partly explained by the “polysemantic” nature of large language models: per-weight importance estimates utilized by EWC-style regularization are too coarse and cannot isolate the knowledge that needs protection. In this paper, we propose regularizing instead in the model’s activation space, using pretrained Sparse Autoencoders (SAEs) as a monosemantic feature dictionary. From the perspective of constrained optimization, we derive a new loss function that uses the SAE feature dictionary to explicitly balance stability and plasticity, and show that EWC is a special case in the one-sided weight-space penalty setting. Unlike replay-based methods that store or revisit examples from earlier tasks, our method requires no previous-task data after mask construction: current-task data is used to compute a compact SAE feature mask, and only this mask is retained for later training. Further, since the feature space has significantly lower dimensionality than the parameter space, the proposed method is more memory efficient. On the TRACE and MedCL continual learning benchmarks, the method achieves the strongest result among approaches without introducing task-specific architectural components, also surpassing traditional weight-space regularization methods like EWC. Beyond performance comparisons, we provide empirical evidence for the polysemanticity thesis: task-relevant representations are linearly separable in the SAE feature basis but indistinguishable from chance in the weight basis, and weight-space protection is nearly non-selective at the concept level. 1 Introduction Large language models (LLMs) increasingly require continual adaptation to new domains, tasks, and interaction formats after deployment. However, sequential fine-tuning can overwrite representations needed for previously learned capabilities, leading to catastrophic forgetting ( 14 , 16 ) . Continual learning methods address this problem through replay, architectural expansion or isolation, and regularization. Among these, regularization is particularly appealing for LLMs because it does not require storing previous-task data, allocating task-specific parameters, or introducing inference-time routing. Most regularization-based continual learning methods protect knowledge at the parameter level. Elastic Weight Consolidation ( 9 , EWC;) , Synaptic Intelligence ( 23 , SI;) , and Memory Aware Synapses ( 1 , MAS;) assign importance scores to individual weights and penalize subsequent changes. This strategy is effective when parameters provide a reasonably selective unit of protection, but it degrades in LLMs. On TRACE, LoRA-based EWC yields only limited improvement over unprotected sequential fine-tuning ( 21 ) ; Our experiments also show that EWC achieves low forgetting primarily by overly reducing plasticity rather than from genuinely protecting acquired knowledge, resulting in weak overall performance. We argue that this failure is structural rather than merely a consequence of imperfect hyperparameter tuning. Mechanistic interpretability offers a plausible explanation through the superposition hypothesis ( 6 ) . Neural networks can encode more features than their dimensionality would normally allow by representing different features in overlapping directions. This gives rise to polysemanticity: individual neurons, and consequently the weights connected to them, can participate in multiple unrelated concepts ( 15 , 2 ) . For weight-space regularization, this creates a mismatch between the unit of importance estimation and the unit of knowledge preservation. Therefore, the diagonal Fisher used by EWC cannot distinguish whether a weight is important for concept A, concept B, or both. Protecting that weight to preserve concept A therefore also constrains concept B, reducing the model’s capacity to learn new tasks. We identify this non-selective protection as a key reason why weight-space regularization fails to scale to LLMs. In this paper, we propose to mitigate the problem by adopting Sparse Autoencoders (SAEs) as a selective coordinate system for continual learning. Trained to reconstruct model activations through sparse overcomplete representations, SAEs decompose dense activations into features that are often more monosemantic than individual neurons ( 7 , 3 ) . A pretrained SAE is used as a fixed feature dictionary. For each task, we estimate which SAE features are activated by task data and construct a task-specific relevance mask, with high-relevance features defining the adaptive region, and low-relevance features defining the protected region. This changes the regularization target from entangled parameters to semantically more selective activation features. We further derive the training objective from a constrained optimization formulation rather than conventionally designing the regularizer in the ad-hoc manner, which simply restricts the model from drifting too much from the original weights. The model should minimize the current-task loss in the SAE feature space satisfying a) protected features should not drift beyond a stability budget, and b) task-relevant features should adapt sufficiently to avoid the degenerate solution of preserving everything but learning nothing, which are defined as stability and plasticity constraints, respectively. Consequently, applying Lagrangian relaxation and a squared-hinge penalty yields two corresponding losses, namely a protect loss and a guide loss. Under this framework, EWC becomes a special case which measures drift in weight space and includes no plasticity constraint. The resulting method requires no previous-task data after mask construction. Current-task examples are used to compute a task-specific SAE feature mask; after this step, only the compact mask is retained, and no examples or activations from that task are stored for later replay. Unlike anchor-based feature distillation methods such as SAE-FD ( 24 ) , which preserve knowledge by storing previous-task anchors or activations and matching them during later training, our method regularizes SAE feature drift using current-task data and the frozen base model, requiring no replay buffer, no stored activations, no per-task parameters, and no inference-time routing. We evaluate SAE-guided activation regularization on TRACE-5000 and MedCL, covering both cross-domain and within-domain continual learning. On TRACE, our method achieves the strongest performance among non-architectural approaches, outperforming weight-space regularizers, gradient-projection methods, and replay-based baselines under matched training conditions. On MedCL, it achieves the best result among methods that retain no previous-task examples. Beyond aggregate performance, we provide mechanistic evidence for our central claim: task-relevant representations are more separable in SAE feature space than in weight space, and protecting weights imposes broader collateral constraints than protecting SAE features. These results show that the effectiveness of regularization depends not only on the penalty strength, but also on the representation space in which the constraint is imposed. Our contributions are summarized as follows: • A practical CL method using SAE features. We introduce activation-space regularization via pretrained SAEs. After mask construction, our method stores only compact SAE feature masks and retains no previous-task examples or anchor activations. • Principled derivation from constrained optimization. We formulate the stability-plasticity tradeoff as a constrained optimization problem in SAE feature space, derive a squared-hinge training loss via Lagrangian relaxation and the quadratic penalty method. • Empirical evidence for the polysemanticity thesis. We demonstrate that superposition undermines weight-space protection through two experiments. A separability test shows that task-relevant representations are linearly separable in the SAE feature basis (AUC 0.88) but indistinguishable from chance in the weight basis (0.50). A collateral-constraint test shows that protecting one task’s weights constrains the next task’s features at 91–96% the rate of its own, whereas feature-space protection reduces this to 43–61%. • Scalability Advantage. Because feature

Read the complete text