Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning

Ishir Garg, Neel Kolhe, Andy Peng, Rohan Gopalam · 2026

Read Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning by Ishir Garg, Neel Kolhe, Andy Peng, Rohan Gopalam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12816.txt.

Introduction

Ishir Garg, Neel Kolhe, Andy Peng, and Rohan Gopalam tackle forgetting in continual learning. Continual learning asks a network to acquire new knowledge task after task. The key challenge is to learn new tasks without catastrophically forgetting old ones. Forgetting can wipe out skills that took long training to build.

Their optimizer enforces Fisher orthogonal constraints on parameter updates. Updates are projected so they do not disturb directions vital to old tasks. The Fisher information marks which directions matter most. Their method is called Fisher orthogonal projected natural gradient descent. It aims to preserve old task performance while learning proceeds. Such optimizers serve systems that must keep learning in the field.

Catalog record

CREATORIshir Garg, Neel Kolhe, Andy Peng, Rohan Gopalam
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-12816

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning Ishir Garg, Neel Kolhe, Andy Peng, Rohan Gopalam Abstract Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tasks. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks. Unlike existing methods that operate in Euclidean parameter space, FOPNG projects gradients onto the Fisher-orthogonal complement of previous task gradients. This approach unifies natural gradient descent with orthogonal gradient methods within an information-geometric framework. We provide theoretical analysis deriving the projected update, describe efficient and practical implementations using the diagonal Fisher, and demonstrate strong results on standard continual learning benchmarks such as Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100. Our code is available at https://github.com/ishirgarg/FOPNG . 1 Introduction Continual learning addresses the fundamental challenge of enabling machine learning models to learn sequentially from non-stationary data streams without forgetting previously acquired knowledge—a problem known as catastrophic forgetting ( 18 ) . When a neural network is trained on a sequence of tasks, gradient updates that improve performance on the current task typically degrade performance on previously learned tasks. The severity of catastrophic forgetting presents a significant barrier to deploying neural networks in realistic scenarios where data arrives sequentially and storage of all past data is impractical. Biological learning systems exhibit remarkable stability-plasticity trade-offs, maintaining existing knowledge while incorporating new information. Achieving similar capabilities in artificial neural networks remains an open challenge that has motivated substantial research. Existing methods for continual learning fall into three broad categories: regularization-based methods that penalize changes to important parameters, replay-based methods that store and rehearse examples from previous tasks, and architecture-based methods that allocate dedicated parameters for each task. While these approaches have achieved varying degrees of success, they share a common limitation: they operate in Euclidean parameter space, which fails to account for the natural geometry of probabilistic models. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, a geometrically principled optimizer that addresses catastrophic forgetting by enforcing orthogonality constraints in the Fisher-Riemannian manifold rather than Euclidean space. The key insight is that the Fisher information matrix provides a coordinate-invariant measure of how parameter changes affect model predictions. By projecting gradients onto the Fisher-orthogonal complement of previous task gradients, FOPNG reduces the impact that updates have on the output distributions of prior tasks while making efficient progress on the current task. Our main contributions are: (1) a theoretical framework that unifies natural gradient descent with orthogonal gradient methods within information geometry, (2) a closed-form solution for the constrained optimization problem defining the projected update, and (3) practical algorithms using efficient Fisher approximations that demonstrate strong empirical results on continual-learning benchmarks. 2 Background and Related Work In this section, we provide a comprehensive background on the foundational concepts underlying FOPNG and situate our work within the broader continual learning literature. We begin with the mathematical foundations of information geometry and natural gradients, then review existing approaches to continual learning, and finally discuss related work on efficient Fisher approximations. 2.1 Information Geometry and the Fisher Information Matrix The Fisher information matrix lies at the heart of our approach, providing the geometric structure that enables principled continual learning. For a parametric model p_{\theta}(y|x) with parameters \theta\in\mathbb{R}^{p} , the Fisher information matrix is F(\theta)=\\ \mathbb{E}_{x\sim q(x)}\mathbb{E}_{y\sim p_{\theta}(\cdot|x)}\nabla_{\theta}\log p_{\theta}(y|x)\nabla_{\theta}\log p_{\theta}(y|x)^{\top} (1) where q(x) is the data distribution ( 2 ) . Intuitively, it represents how fast a probability distribution changes at a point with respect to small perturbations in its parameter. The Fisher matrix has several important properties that make it fundamental to our approach. The Fisher matrix defines a Riemannian metric on the parameter manifold, turning the space of probability distributions into a curved geometric space. Under this metric, the infinitesimal distance between p_{\theta} and p_{\theta+d\theta} is ds^{2}=d\theta^{\top}F(\theta)d\theta . This geometric perspective was developed extensively by Amari and others in the field of information geometry ( 2 ) . A powerful property of the Fisher metric is that it is invariant under reparameterization. Theorem 2.1 (Reparameterization Invariance of Fisher Norm) . Let \phi=\phi(\theta) be a smooth bijection with Jacobian J=\frac{\partial\phi}{\partial\theta} . Let F_{\theta} and F_{\phi} denote the Fisher information matrices in the \theta and \phi parameterizations, respectively. Then for any tangent vector \delta\theta , \|\delta\theta\|_{F_{\theta}}=\|\delta\phi\|_{F_{\phi}} (2) where \delta\phi=J\delta\theta . Proof. See Appendix A.1 . ∎ Intuitively, this means that optimization in the Fisher metric respects the intrinsic structure of the probability model rather than depending on arbitrary choices of parameterization. Euclidean gradient descent, on the contrary, is not reparameterization invariant and can behave very differently under coordinate changes. Another key property of the Fisher information for FOPNG is that the Fisher matrix locally approximates the KL divergence between nearby distributions. Theorem 2.2 (Local Quadratic Approximation of KL Divergence) . For a small parameter step v , the KL divergence satisfies \text{KL}(p_{\theta}\|p_{\theta+v})=\frac{1}{2}v^{\top}F_{\theta}v+o(\|v\|^{2}). (3) Proof. See Appendix A.2 . ∎ This means that the Fisher norm \|v\|_{F}=\sqrt{v^{\top}Fv} directly quantifies the change in model predictions induced by a parameter update. Steps that are small in Fisher norm cause small changes in the output distribution, regardless of their Euclidean magnitude. 2.2 Continual Learning: Problem Formulation and Existing Approaches In continual learning, we seek to learn a sequence of tasks \{T_{1},T_{2},\ldots,T_{n}\} without catastrophic forgetting. At each stage, the model parameterized by \theta must adapt to a new task T_{k} with loss L_{k}(\theta) while preserving performance on all previous tasks T_{1},\ldots,T_{k-1} . Standard gradient descent on L_{k} typically causes parameters to drift in directions that degrade previously learned representations. The fundamental tension is between plasticity —the ability to learn new information—and stability —the preservation of old knowledge. Different methods approach this trade-off through different mechanisms, which we now review. 2.3 Natural Gradient Descent Natural gradient descent exploits Fisher geometry for efficient optimization ( 2 ) . Rather than Euclidean-small steps, it takes Fisher-small steps: \theta_{t+1}=\theta_{t}-\eta F(\theta_{t})^{-1}\nabla_{\theta}L(\theta_{t}), (4) derived by minimizing the loss’s linear approximation subject to a KL divergence constraint. Natural gradient methods converge faster than standard gradient descent by accounting for loss landscape curvature ( 17 ) , and reparameterization invariance makes them robust to architecture choices affecting parameter scaling. 2.4 Regularization-Based Methods Regularization methods penalize changes to task-important parameters. Elastic Weight Consolidation (EWC) ( 9 ) adds: \Omega(\theta)=\frac{\lambda}{2}\sum_{i}F_{i}(\theta_{i}-\theta_{i}^{*})^{2}, (5) where diagonal Fisher entries F_{i} identify important parameters at learned parameters \theta^{*} . While pioneering Fisher information use in continual learning, EWC’s diagonal approximation ignores correlations, and performance is highly sensitive to hyperparameter tuning ( 11 ) . FOPNG achieves more stable performance across hyperparameters, requiring minimal tuning. Synaptic Intelligence ( 22 ) estimates parameter importance online during training. Memory Aware Synapses ( 1 ) learns importance unsupervised. These also share the regularization paradigm. 2.5 Replay-Based Methods Replay methods store and rehearse previous task examples. Experience Replay maintains a buffer sampled during training ( 4 ) . Gradient Episodic Memory (GEM) ( 15 ) projects gradients to prevent loss increases on stored examples. A-GEM ( 3 ) efficiently projects onto a single reference gradient. These show strong performance when memory permits storing representative examples. GEM’s constraint formulation relates to

Read the complete text