Continual Learning with Quasi-Newton Methods
STEVEN VANDER EECKT, HUGO VAN HAMME · 2025
Read Continual Learning with Quasi-Newton Methods by STEVEN VANDER EECKT, HUGO VAN HAMME (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19939.txt.
Introduction
Steven Vander Eeckt and Hugo Van Hamme present a 2025 paper on teaching neural networks new tasks without erasing old ones, a failure called catastrophic forgetting. A popular guard, Elastic Weight Consolidation, pins down important weights, but it estimates importance crudely, treating every weight as if it acted alone. Real weights interact, so the guard often misjudges.
Their method samples richer curvature estimates, borrowed from quasi-Newton optimization, the family of tools that track how weights lean on each other. The richer map guards knowledge better without any change to the network's shape. Across four benchmark suites, their approach beat the older guard consistently at keeping old skills while learning new ones.
Catalog record
| CREATOR | STEVEN VANDER EECKT, HUGO VAN HAMME |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19939
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Continual Learning with Quasi-Newton Methods STEVEN VANDER EECKT, HUGO VAN HAMME Abstract Catastrophic forgetting remains a major challenge when neural networks learn tasks sequentially. Elastic Weight Consolidation (EWC) attempts to address this problem by introducing a Bayesian-inspired regularization loss to preserve knowledge of previously learned tasks. However, EWC relies on a Laplace approximation where the Hessian is simplified to the diagonal of the Fisher information matrix, assuming uncorrelated model parameters. This overly simplistic assumption often leads to poor Hessian estimates, limiting its effectiveness. To overcome this limitation, we introduce Continual Learning with Sampled Quasi-Newton (CSQN), which leverages Quasi-Newton methods to compute more accurate Hessian approximations. CSQN captures parameter interactions beyond the diagonal without requiring architecture-specific modifications, making it applicable across diverse tasks and architectures. Experimental results across four benchmarks demonstrate that CSQN consistently outperforms EWC and other state-of-the-art baselines, including rehearsal-based methods. CSQN reduces EWC’s forgetting by 50 % \% and improves its performance by 8 % \% on average. Notably, CSQN achieves superior results on three out of four benchmarks, including the most challenging scenarios, highlighting its potential as a robust solution for continual learning. † † history: Received 17 February 2025, accepted 5 March 2025, date of publication 13 March 2025, date of current version 21 March 2025. † † doi: 10.1109/ACCESS.2025.3551146 † † address: Department Electrical Engineering ESAT-PSI, KU Leuven, B-3001 Leuven, Belgium † † corresponding: Corresponding author: Steven Vander Eeeckt (e-mail: [email protected]). I Introduction Since the 2010s, Artificial Neural Networks (ANNs) have been able to match or even surpass human performance on a wide variety of tasks. However, when presented with a set of tasks to be learned sequentially—a setting referred to as Continual Learning (CL)—ANNs suffer from catastrophic forgetting [ 1 ] . Unlike humans, ANNs struggle to retain previously learned knowledge when extending their knowledge. Naively adapting an ANN to a new task generally leads to a deterioration in the network’s performance on previous tasks. Many CL methods have been proposed to alleviate catastrophic forgetting. One of the most well-known is Elastic Weight Consolidation (EWC) [ 2 ] , which approaches CL from a Bayesian perspective. After training on a task, EWC uses Laplace approximation [ 3 ] to estimate a posterior distribution over the model parameters for that task. When training on the next task, this posterior is used via a regularization loss to prevent the model from catastrophically forgetting the previous task. To estimate the Hessian, which is needed in the Laplace approximation to measure the (un)certainty of the model parameters, EWC uses the Fisher Information Matrix (FIM). Furthermore, to simplify the computation, EWC assumes that the FIM is approximately diagonal. EWC thus uses the diagonal of the FIM to estimate the Hessian. While EWC mitigates catastrophic forgetting, its effectiveness is limited by its assumption of an approximately diagonal FIM. This assumption oversimplifies parameter correlations, leading to poor Hessian estimates and suboptimal posterior approximations. These inaccuracies might cause EWC to struggle in complex scenarios with significant parameter interactions, severely impacting its performance in continual learning. Hence, we propose Continual Learning with Sampled Quasi-Newton (CSQN), a method designed to overcome the limitations of EWC’s diagonal Hessian approximation. CSQN leverages Quasi-Newton (QN) methods [ 4 ] . QN methods, being second-order optimization algorithms, involve (inverse) Hessian approximations that are updated at each iteration (without being explicitly computed) to find local optima. Two such methods are BFGS (Broyden, Fletcher, Goldfarb, Shanno) and SR1 (Symmetric Rank-1), which apply rank-2 and rank-1 updates, respectively, to the Hessian approximation at each iteration. Since our primary interest is in the Hessian approximations from QN methods, training the model with QN methods solely to obtain these approximations would be impractical. Therefore, we propose using Sampled Quasi-Newton (SQN) methods [ 5 ] . In SQN methods, the Hessian approximation is obtained by sampling around the current estimate of the local optimum, meaning that the Hessian is computed anew at each iteration without relying on previous ones. Furthermore, to enhance our method and make it a direct extension of EWC, we use EWC’s Hessian estimate, namely the diagonal of the FIM, as the initial Hessian approximation. Using SQN, we then improve this Hessian approximation, which is no longer restricted to being diagonal. Similar to EWC, this improved Hessian approximation is used in the regularization loss to prevent forgetting previous tasks while learning new ones. We evaluate our proposed method through four benchmarks, comparing it to the most relevant and competitive baselines. These include EWC, which our method extends and improves, as well as other CL methods, including those relying on the storage of past data. Our results show that CSQN consistently outperforms these baselines across all benchmarks, demonstrating superior performance after learning multiple tasks in each experiment. Additionally, we enhance our method by proposing mechanisms to reduce the memory requirements of CSQN and evaluate their impact. Our findings indicate that the performance loss associated with these memory-efficient versions is limited, suggesting that CSQN can achieve a balance between effectiveness and memory usage. This paper is structured as follows. We start by reviewing related work in Section II . Next, in Section III , we explain our proposed method (Section III-E ) after a short review of EWC (Section III-B ) and (Sampled) Quasi-Newton methods (Section III-D ). In Section IV , we present the experimental results and compare our method to EWC and other baselines. Section V discusses the strengths and challenges of the proposed CSQN method. Finally, Section VI concludes the paper. II Related Work Continual Learning (CL) methods are generally grouped into three main categories: regularization-based, rehearsal-based, and parameter-isolation methods [ 6 ] . Each of these categories tackles catastrophic forgetting using distinct approaches to retain knowledge of previously learned tasks. For a more extensive overview of recent advancements and categorizations in Continual Learning, readers are encouraged to consult recent comprehensive reviews such as [ 7 , 8 ] . The following discussion focuses on methods most relevant to our work. II-A Regularization-Based Methods To prevent catastrophic forgetting, regularization-based methods regularize the learning of new tasks. Following [ 6 ] , two classes of regularization-based methods are prior-focused methods and data-focused methods. EWC, using the diagonal of the FIM to estimate the importance of parameters to previous tasks, belongs to the prior-focused methods. Similar to EWC are Synaptic Intelligence (SI) [ 9 ] and Memory-Aware Synapses (MAS) [ 10 ] , which compute the importance weights of the model parameters differently. While SI estimates the importance of parameters by accumulating each parameter’s contribution to the reduction in the loss, MAS considers the gradients of the squared norm of the model’s output with respect to each parameter. Similar work to EWC, MAS and SI includes [ 11 , 12 , 13 ] . Many works build on EWC’s ideas. Online EWC [ 14 ] allows EWC to be used in an online way, without requiring task boundaries. [ 15 ] proposes Riemanian Walk, which combines the regularization losses of Online EWC and SI. Moreover, to address EWC’s assumption of an approximately diagonal FIM, R-EWC (Rotated-EWC) [ 16 ] rotates the parameter space layer-wise such that the corresponding FIM is indeed approximately diagonal. While R-EWC improves the performance of EWC when adapting