Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models
Zekun Wang, Anant Gupta, Zihan Dong, Christopher J. MacLellan · 2025
Read Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models by Zekun Wang, Anant Gupta, Zihan Dong, Christopher J. MacLellan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23593.txt.
Introduction
Neural networks that learn task after task tend to forget the early ones. Zekun Wang and 3 co-authors attack that forgetting with a fact about diffusion models: at low signal-to-noise levels, per-sample gradients line up, so the Fisher matrix is nearly rank one. Elastic weight consolidation normally guards old knowledge with a diagonal Fisher guess, which misses that main direction. Their rank-1 variant costs about the same yet tracks the true curvature better.
Paired with replay of generated samples, the penalty held knowledge across tasks while limiting drift. On image generation sets from MNIST to ImageNet, forgetting fell sharply against replay-only and diagonal baselines, and nearly vanished on the two simplest sets.
Catalog record
| CREATOR | Zekun Wang, Anant Gupta, Zihan Dong, Christopher J. MacLellan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23593
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Models Zekun Wang, Anant Gupta, Zihan Dong, Christopher J. MacLellan Abstract Catastrophic forgetting remains a central obstacle for continual learning in neural models. Popular approaches—replay and elastic weight consolidation (EWC)—have limitations: replay requires a strong generator and is prone to distributional drift, while EWC implicitly assumes a shared optimum across tasks and typically uses a diagonal Fisher approximation. In this work, we study the gradient geometry of diffusion models, which can already produce high-quality replay data. We provide theoretical and empirical evidence that, in the low signal-to-noise ratio (SNR) regime, per-sample gradients become strongly collinear, yielding an empirical Fisher that is effectively rank-1 and aligned with the mean gradient. Leveraging this structure, we propose a rank-1 variant of EWC that is as cheap as the diagonal approximation yet captures the dominant curvature direction. We pair this penalty with a replay-based approach to encourage parameter sharing across tasks while mitigating drift. On class-incremental image generation datasets (MNIST, FashionMNIST, CIFAR-10, ImageNet-1k), our method consistently improves average FID and reduces forgetting relative to replay-only and diagonal-EWC baselines. In particular, forgetting is nearly eliminated on MNIST and FashionMNIST and is roughly halved on ImageNet-1k. These results suggest that diffusion models admit an approximately rank-1 Fisher. With a better Fisher estimate, EWC becomes a strong complement to replay: replay encourages parameter sharing across tasks, while EWC effectively constrains replay-induced drift. 1 Introduction The task of continual learning aims to train neural models on a stream of tasks without revisiting the full past data. A long–standing obstacle is catastrophic forgetting—when learning new tasks drastically degrades performance on earlier ones ( McCloskey & Cohen 1989 ; French 1999 ; Parisi et al. 2019 ) . In contrast, humans exhibit a striking robustness to interference, partly due to systems-level mechanisms such as memory consolidation and replay in the hippocampal–neocortical loop ( McClelland et al. 1995 ; McGaugh 2000 ; Foster & Wilson 2006 ) . These observations have inspired a family of continual learning methods that explicitly encode consolidation or replay in training deep networks on continuous tasks. Two representative approaches are elastic weight consolidation (EWC) ( Kirkpatrick et al. 2017 ) and generative replay ( Shin et al. 2017 ) . EWC constrains parameter changes with a quadratic penalty weighted by an estimate of the Fisher information at a previously learned task, constraining updates to remain near parameter directions that support old tasks ( Kirkpatrick et al. 2017 ; Schwarz et al. 2018 ) . Intuitively, EWC behaves like an online consolidation process that selectively “stiffens” important parameters. Generative replay maintains a generator and distills samples from past tasks while learning new ones, thereby approximating rehearsal without storing original data ( Shin et al. 2017 ; van de Ven & Tolias 2018 ) . However, replay inherits the generator’s imperfections and can amplify distributional shift. EWC in practice relies on a diagonal Fisher approximation, which neglects cross-parameter correlations and struggles to find a shared parameter space between tasks in overparameterized models, particularly when tasks have disjoint optimum. These observations suggest a complementary pairing: replay can encourage parameter sharing by exposing shared data support, while a stronger approximation to Fisher information can constrain updates and mitigate replay’s residual shift. In this work, we study the gradient behavior of diffusion models ( Ho et al. 2020b ; Nichol & Dhariwal 2021 ) , which are already capable of generating high quality replay data. Our starting point is the observation that diffusion models admit a tractable gradient structure when the signal-to-noise ratio (SNR) is lower (at later timesteps). As a model converges, the per-sample gradients g become approximately collinear with their mean u . This makes the empirical Fisher F effectively rank-1: F\;=\;\mathbb{E}\big[g\,g^{\top}\big]\;\approx\;\alpha\,u\,u^{\top},\quad u=\mathbb{E}[g], This yields a consolidation penalty that captures the dominant curvature direction “for free” from model gradients. In contrast, the commonly used diagonal Fisher approximation captures almost no curvature when SNR is low. We leverage this structure to instantiate a rank-1 EWC that complements generative replay: replay encourages cross-task parameter overlap while our proposed rank-1 EWC constrains updates along the principal sensitive direction to the shared optimum across tasks. Our main contributions are: (1) We provide both theoretical and empirical characterizations of Fisher information geometry in diffusion models, showing that low SNR induces a near rank-1 Fisher aligned with the mean gradient. (2) We propose a practical rank-1 EWC penalty that is as cheap as a diagonal penalty but captures more curvature information for diffusion models. (3) We demonstrate that combining rank-1 EWC with distillation-based replay substantially reduces forgetting in continual image generation tasks, improving generation fidelity and stability across long horizons. On MNIST, FashionMNIST, CIFAR-10, and ImageNet-1k, our method consistently outperforms replay-only and diagonal Fisher baselines. In particular, forgetting is nearly eliminated on MNIST and FashionMNIST and is roughly halved on ImageNet-1k, the longest-horizon setting, relative to baselines. 2 Background and related work 2.1 Diffusion models Diffusion models are a family of generative models that define a fixed forward noising process and learn a reverse (denoising) process that maps Gaussian noise to data ( Sohl-Dickstein et al. 2015 ; Ho et al. 2020a ) . The forward chain corrupts data x_{0}\sim q_{0} via a Markov process q(x_{t}\mid x_{t-1})\;\sim\;\mathcal{N}\!\big(\sqrt{1-\beta_{t}}\,x_{t-1},\,\beta_{t}\mathbf{I}\big),\quad q(x_{t}\mid x_{0})\;\sim\;\mathcal{N}\!\big(\sqrt{\bar{\alpha}_{t}}\,x_{0},\,(1-\bar{\alpha}_{t})\mathbf{I}\big), where \alpha_{t}=1-\beta_{t} , \bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s} , \beta_{t}\in(0,1) is the noise level, and t\in\{0\cdots N\} is the discrete forward noising process timestep. The reverse transitions p_{\theta}(x_{t-1}\!\mid\!x_{t}) are parameterized by a neural network \varepsilon_{\theta} and trained by maximizing a variational lower bound (ELBO). In practice, the ELBO reduces to a reweighted denoising loss ( Ho et al. 2020a ) : \mathcal{L}_{\text{simple}}(\theta)\;=\;\frac{1}{2}\mathbb{E}_{t,\,x_{0},\,\varepsilon\sim\mathcal{N}(0,\mathbf{I})}\Big[\;\big\|\varepsilon-\varepsilon_{\theta}(x_{t},t)\big\|_{2}^{2}\;\Big],\quad x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\,\varepsilon. This surrogate loss can be interpreted as a score-based generative modeling that estimates the score \nabla_{x_{t}}\log q_{t}(x_{t}) at each timestep, where q_{t}(x_{t})=\int q(x_{t}\mid x_{0})q_{0}(x_{0})dx_{0} , and then integrates a reverse-time SDE/ODE to sample ( Song & Ermon 2019 ; Song et al. 2021b ; Vincent 2011 ; Hyvärinen 2005 ) : \nabla_{x_{t}}\log q_{t}(x_{t})=-\frac{1}{\sqrt{1-\bar{\alpha}_{t}}}\,\mathbb{E}[\varepsilon\mid x_{t}], and the model’s score estimate follows from s_{\theta}(x_{t},t)\;=\;-\tfrac{1}{\sqrt{1-\bar{\alpha}_{t}}}\,\varepsilon_{\theta}(x_{t},t) , where \varepsilon_{\theta}(x_{t},t)=\mathbb{E}[\varepsilon\mid x_{t}] at model optimum. Denoising Diffusion Implicit Models ( Song et al. 2021a ) further show that one can define a non-Markovian, deterministic sampling path that preserves DDPM’s per-time marginal distributions while allowing far fewer steps. In this work, we use a DDIM sampling process for faster image generation. 2.2 Elastic weight consolidation Elastic weight consolidation mitigates catastrophic forgetting by regularizing parameter updates using information about how important each weight was to previously learned tasks ( Kirkpatrick et al. 2017 ) . It casts continual learning as approximate Bayesian updating: for tasks 1{:}T , the posterior factors as \;p(\theta\mid\mathcal{D}_{1:T})\propto p(\mathcal{D}_{T}\mid\theta)\,p(\theta\mid\mathcal{D}_{1:T-1}) . EWC approximates the previous-task posterior p(\theta\mid\mathcal{D}_{1:T-1}) with a Laplace approximation around the prior optimum \theta^{\star}_{T-1} , yielding -\log p(\theta\mid\mathcal{D}_{1:T-1})~\approx~\frac{1}{2}\,(\theta-\theta^{\star}_{T-1})^{\top}F^{(T-1)}(\theta-\theta^{\star}_{T-1}), where F^{(T-1)} is the Fisher information evaluated at \theta^{\star}_{T-1} . Plugging this quadratic surrogate into the negative log-posterior for task k gives the EWC objective \mathcal{L}_{\text{EWC}}(\theta)~=~\mathcal{L}_{T}(\theta)\;+\;\frac{\lambda}{2}\sum_{k=1}^{T-1}(\theta-\theta^{\star}_{k})^{\top}F^{(k)}(\theta-\theta^{\star}_{k}), (1) with \lambda trading off plasticity and stability. Intuitively, parameters with high curvature under the previous task are penalized more strongly, discouraging changes that would degrade old performance. In the original work, \mathcal{L}_{T}(\theta) is the negative log-likelihood for a classification model. For latent-variable generative models such as diffusion models, one can replace this term with a tractable variational