Natural Gradient Descent for Online Continual Learning
Joe Khawand, David Colliaux · 2026
Read Natural Gradient Descent for Online Continual Learning by Joe Khawand, David Colliaux (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20898.txt.
Introduction
Khawand and Colliaux bring natural gradient descent to online continual learning for image classification. In this setting, models learn from a stream of images and can forget old tasks. Fast learning is hard without that forgetting. Their method uses a Kronecker factored approximation of the Fisher information matrix. It boosts results across several existing continual learning methods. Gains are largest when it is paired with known training tricks. Tests cover Split CIFAR-100, CORE50, and Split miniImageNet.
Catalog record
| CREATOR | Joe Khawand, David Colliaux |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36245 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20898
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Natural Gradient Descent for Online Continual Learning Joe Khawand, David Colliaux Abstract Online Continual Learning (OCL) for image classification represents a challenging subset of Continual Learning, focusing on classifying images from a stream without assuming data independence and identical distribution (i.i.d). The primary challenge in this context is to prevent catastrophic forgetting, where the model’s performance on previous tasks deteriorates as it learns new ones. Although various strategies have been proposed to address this issue, achieving rapid convergence remains a significant challenge in the online setting. In this work, we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC). This method demonstrates substantial improvements in performance across all OCL methods, particularly when combined with existing OCL tricks, on datasets such as Split CIFAR-100, CORE50, and Split miniImageNet. 1 Introduction With the surge of embedded AI systems and the growing need for them, Continual Learning presents itself as an elegant solution to on-device learning. This paradigm refers to the ability of a model to learn by continuously assimilating knowledge over time as observed in biological entities 21 . Mathematically, this refers to the ability to train a model while breaking the i.i.d. assumption on the data, sending classes one after the others, for example, in the classification scenario. Naively training a model in this scenario leads to the phenomenon of catastrophic forgetting 11 ; 29 , where the model forgets previously learned representations and specializes on the last seen task. Most of the research focuses on scenarios in which the model can iterate for multiple epochs 39 and use batches of reasonable size. Unfortunately, doing so requires significant storage that is more often than not unavailable on devices. Online Continual Learning (OCL) 25 ; 37 considers the more realistic but more difficult scenario where the model can only iterate once on the current task and has no access to the previous ones. The model is thus required to achieve satisfactory performance from a single pass over the online data stream using very small batch sizes. In this context, the model may encounter new categories (Online Class Incremental, OCI ) or changes in the characteristics of the data such as alterations in the background, the introduction of blur or noise, changes in lighting, or the presence of obstructions (Online Domain Incremental, ODI ) 39 . We focus our attention on the methods that do not require access to task labels, as this does not represent a realistic OCL scenario. In this complex scenario, OCL methods struggle to achieve high accuracies as shown in the survey 25 . There is still a big gap between OCL-trained models and Offline trained models. In this paper, we explore the use of 2 nd order optimization in Online Continual Learning. Motivated by the need for faster convergence, we investigate the use of the Natural Gradient Descent optimizer 5 with Kronecker Factored Approximate Curvature (KFAC) 26 , in the OCL scenario. We demonstrate that this method, presented as the Steepest Descent in the distribution space, boasts increased performance across all the OCL methods and tricks presented in 25 , on datasets such as Split CIFAR-100 20 , CORE50 23 , and Split MiniImageNet 25 . Figure 1: Average End Accuracy on Split-CIFAR-100 20 for 3 buffer sizes (1k-5k-10k) using ER 33 in combination with OCL tricks 25 . Our paper makes the following contributions: 1. We propose the use of Natural Gradient descent with Kronecker Factored Approximate Curvature (NGD-KFAC) as an optimization method in Online Continual Learning (OCL) for image classification. This is a novel application of NGD-KFAC in the OCL domain. 2. Our experiments show that NGD-KFAC significantly improves the performance of OCL models on various OCI and ODI benchmarks. This method enables most existing online continual learning techniques to achieve better results with only a minor change to the optimizer, highlighting its effectiveness in reducing catastrophic forgetting and enhancing model convergence. 2 Related work Continual learning has been studied to address the dynamic nature of real-world data, where the assumption of a static data distribution often does not hold 39 . A considerable portion of this research has been dedicated to mitigating the phenomenon of catastrophic forgetting 11 ; 29 , a significant challenge when models are sequentially trained on multiple tasks. A comprehensive survey by Mai et al. (2022) 25 highlights various strategies used in Online Continual Learning (OCL), showing the complexity of training models that are constrained to learn from data streams with limited passes and without access to previous tasks. This survey shows the gap between traditional offline learning and the difficult requirements of OCL scenarios. The use of second-order optimization methods, specifically Natural Gradient Descent (NGD), has shown promise in machine learning for its ability to account for the underlying data distribution’s geometry. Amari (1998) 5 introduced NGD, leveraging the Fisher Information Matrix (FIM) to guide optimization in a manner that respects the information geometry of parameter space, offering a more principled update path compared to first-order methods like Stochastic Gradient Descent (SGD). Despite its theoretical advantages, the practical application of NGD in deep learning has been limited by computational challenges, particularly in calculating and inverting the FIM for large models. To overcome these challenges, Martens and Grosse (2015) 26 proposed the Kronecker-Factored Approximate Curvature (KFAC) method, an approximation of NGD that reduces the computational overhead by approximating the FIM with a Kronecker product. EKFAC, introduced by George et al. (2018) 13 , enhances the KFAC approximation by rescaling the Kronecker factors with a diagonal matrix derived from singular value decomposition (SVD), aiming to mitigate some of these challenges. Conversely, the KBFGS 14 method seeks to approximate the inverse of the Kronecker factors through low-rank Broyden-Fletcher-Goldfarb-Shanno (BFGS) updates, offering a different approach to handling the inversion problem. The TENGraD method, developed by Soori et al. 36 , is accurate, fast, and needs less memory because it inverts blocks of the Fisher Information Matrix (FIM). It does this by efficiently breaking down the FIM and reusing certain calculations. Despite the advancements offered by EKFAC, KBFGS, and TenGrad, our tests showed that KFAC was more numerically stable, especially for our specific needs. We found KFAC to be more reliable and effective in this OCL setting, which is why we preferred it. Works 42 ; 2 ; 19 ; 22 have explored the potential of second-order methods and regularization techniques to mitigate catastrophic forgetting. These methods provide a promising direction for improving model robustness and adaptability in dynamic learning environments, but they still fall short of other methods like experience replay 33 , especially in the OCL scenario 25 . Additionally, Variational Continual Learning (VCL) 31 ; 38 has been proposed as an improvement of those methods 19 ; 22 . However, to our knowledge, there are few direct comparisons of VCL with other methods in OCL settings, and as such, we do not include VCL in our evaluations. This paper builds upon these works by introducing an application of NGD-KFAC, within the OCL framework. Our approach leverages the strengths of second-order optimization to address the challenges of OCL, such as rapid convergence