Analysis of Overparameterization in Continual Learning under a Linear Model
\name Daniel Goldfarb \email [email protected], \name Paul Hand \email [email protected] · 2025
Read Analysis of Overparameterization in Continual Learning under a Linear Model by \name Daniel Goldfarb \email [email protected], \name Paul Hand \email [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10442.txt.
Introduction
Daniel Goldfarb and Paul Hand prove that oversized models can protect old skills while learning new ones, in a 2025 paper. Neural networks trained on tasks in sequence tend to forget catastrophically. Most defenses add explicit machinery against forgetting. The authors ask whether sheer size alone, called overparameterization, can do the job in a stripped-down setting.
They analyze plain gradient descent on a linear regression model facing two tasks in a row. As the parameter surplus grows large enough, the twice-trained model still performs well on the first task. The proof also yields a fresh risk bound for single-task regression, of interest to double-descent theory. The result gives continual learning a cleaner theoretical foothold: in simple worlds, capacity itself is a memory aid.
Catalog record
| CREATOR | \name Daniel Goldfarb \email [email protected], \name Paul Hand \email [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-10442
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Analysis of Overparameterization in Continual Learning under a Linear Model \name Daniel Goldfarb \email [email protected], \name Paul Hand \email [email protected] Abstract Autonomous machine learning systems that learn many tasks in sequence are prone to the catastrophic forgetting problem. Mathematical theory is needed in order to understand the extent of forgetting during continual learning. As a foundational step towards this goal, we study continual learning and catastrophic forgetting from a theoretical perspective in the simple setting of gradient descent with no explicit algorithmic mechanism to prevent forgetting. In this setting, we analytically demonstrate that overparameterization alone can mitigate forgetting in the context of a linear regression model. We consider a two-task setting motivated by permutation tasks, and show that as the overparameterization ratio becomes sufficiently high, a model trained on both tasks in sequence results in a low-risk estimator for the first task. As part of this work, we establish a non-asymptotic bound of the risk of a single linear regression task, which may be of independent interest to the field of double descent theory. Editor: Keywords: Continual Learning, Catastrophic Forgetting, Deep Learning Theory, Double Descent, Regularization 1 Introduction The field of lifelong learning aims to develop machine learning systems that can continually learn many tasks by transferring and retaining knowledge from previously learned tasks without the need for retraining ( Chen and Liu (2022) ). This is an important problem because it can be a waste of computation to retrain neural networks from scratch. One challenge of continual learning is the effect of catastrophic forgetting, where models abruptly lose performance on previously learned tasks when training on new ones ( French (1999) ). The amount of expected catastrophic forgetting is affected by multiple aspects of the tasks, like the neural network architecture and task similarity or difficulty. Hence the field of continual learning aims to develop dedicated approaches to mitigate catastrophic forgetting. These include regularization-based methods, architectural modifications, and memory-replay methods ( Li and Hoiem (2017) ; Kirkpatrick et al. (2017) ; Zenke et al. (2017) ; Shin et al. (2017) ). In order to build an understanding of state-of-the-art continual learning methods, it is a prerequisite to understand simpler methods. One such simple method is naively modifying the overparameterization level. Modern neural networks typically operate in the extremely overparameterized regime, where the number of learned parameters is much larger than the number of samples in the training set. In practice overparameterized models perform better than their underparameterized counterparts ( Nakkiran et al. (2021) ; Zhang et al. (2021) ). This phenomenon, coined double descent, sparked active research to explain this behavior ( Adlam and Pennington (2020) ; Blanc et al. (2020) ). While most double descent works contribute to the single-task setting, one continual learning related work ( Mirzadeh et al. (2022) ) ran experiments on popular rotated image benchmarks to show that overparameterized neural networks may be less prone to forgetting than their moderately parameterized counterparts. Goldfarb and Hand (2023) replicated these results for random data permutation tasks. Mathematical theory is necessary in order to understand when continual learning algorithms are needed and when to expect performance degradation in practice. This theory is challenging because the theoretical analysis of trained neural networks is still a nascent field, and additional architectural and algorithmic modifications would increase the challenge of that analysis. As a foundation for understanding the algorithmic developments of continual learning, we aim to understand the simplest setting of gradient descent with linear models modified only by their overparameterization level ( Evron et al. (2023) ; Lin et al. (2023) ). This setting has been previously studied by theoretical papers for double descent in the single-task setting ( Muthukumar et al. (2020) ; Belkin et al. (2020) ; Bartlett et al. (2020) ; Hastie et al. (2022) ). As summarized in Dar et al. (2021) , these works concurrently found that a linear model can enjoy optimal performance in the overparameterized regime when the effective dimensionality of the data is lower than the number of training data samples, and the true dimensionality of the data is higher than the number of training data samples. We aim to leverage existing single-task work as a foundation to quantify the effect of overparameterization in the multi-task setting. The model presented in Hastie et al. (2022) has a strong connection to neural networks, which we fully describe in Section 2 , and we take inspiration from this model in our work. In this paper, we study a two-task continual learning setting with a latent space data model inspired by Hastie et al. (2022) . The tasks will be related by a random orthogonal transformation. This two-task setup is a mathematical idealization of the standard continual learning benchmark where tasks are related by a random permutation ( Kirkpatrick et al. (2017) ). We demonstrate that, under this linear model, if overparameterization is sufficiently high and there are sufficiently many training examples relative to the model’s latent dimensionality, then this two-task setup has low risk on task A both before and after training on task B. We now present an informal version of our main conclusion. Consider tasks A and B each with n training examples and p observed features but only d latent features. We are interested in the overparameterized regime where d<n<p . The data of each task is related by a random p\times p orthogonal matrix. Given task A with data X_{A}\in\mathbb{R}^{n\times p} , task B is given by X_{B}=X_{A}O^{\top} . Both tasks have shared labels y\in\mathbb{R}^{n} as is standard with permutation tasks. Our training procedure will fit task A by minimizing the square loss from initialization 0, and then subsequently training on task B with no explicit mechanism to prevent forgetting. Let \hat{\beta}_{A} be the parameters of the model that is trained on task A and \hat{\beta}_{BA} be the parameters of the model that is subsequently trained on task B. Our main result establishes non-asymptotic bounds with high probability on the risk of task A of several estimators. We informally present it as follows: \displaystyle R(\hat{\beta}_{A}) \displaystyle\lesssim\sqrt{\frac{d}{n}}+\frac{n}{p}, (1) \displaystyle R(\hat{\beta}_{BA}) \displaystyle\lesssim\sqrt{\frac{d}{n}}+\sqrt{\frac{n}{p}}, (2) \displaystyle\frac{R(\hat{\beta}_{BA})-R(\hat{\beta}_{A})}{R(0)-R(\hat{\beta}_{A})} \displaystyle\lesssim\frac{\sqrt{\frac{n}{p}}}{1-C\left(\sqrt{\frac{d}{n}}+\frac{n}{p}\right)}, (3) where R(\cdot) denotes the risk on task A and C is a constant. Equation ( 1 ) quantifies the performance of an estimator trained on a single task. Equation ( 2 ) quantifies the performance on task A of the estimator that trains on task A then task B in sequence. Equation ( 3 ) bounds the ratio between the forgetting due to training on task B relative to the amount of initial learning of task A. Observe that all three bounds are small when the number of latent features is much smaller than the number of training examples, and the number of training examples is much smaller than the number of features. Hence task A is well-learned initially, task A remains well-learned after training on task B, and the forgetting due to training on task B is insignificant in comparison to the amount of learning on task A when initialized at 0. These conditions are the same as the ones summarized by Dar et al. (2021) to describe when double descent occurs for the single-task case. The formal version of this result is presented