Leveraging Lightweight Generators for Memory Efficient Continual Learning

Christiaan Lamers 1, Ahmed Nabil Belbachir 1, Thomas Bäck 2, Niki van Stein 2 · 2025

Read Leveraging Lightweight Generators for Memory Efficient Continual Learning by Christiaan Lamers 1, Ahmed Nabil Belbachir 1, Thomas Bäck 2, Niki van Stein 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19692.txt.

Introduction

Christiaan Lamers, Ahmed Nabil Belbachir, Thomas Bäck, and Niki van Stein shrink the memory that lifelong learning systems need, in a 2025 paper. A model that learns task after task tends to forget the old ones, a failure called catastrophic forgetting. Keeping every old example fixes the forgetting but costs huge storage. The goal is to store as little as possible while keeping the most useful information.

The authors propose tiny generators built from a classic compression method, singular value decomposition. Each generator stores a compact sketch of past data and can produce stand-in examples when old tasks are rehearsed. The sketches need no training, just one fast fitting pass, yet they capture a data distribution from few samples. Plugged into two known methods, experience replay and A-GEM, the generators raised average accuracy in tests while cutting memory sharply.

Catalog record

CREATORChristiaan Lamers 1, Ahmed Nabil Belbachir 1, Thomas Bäck 2, Niki van Stein 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-19692

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Lightweight Generators for Memory Efficient Continual Learning Christiaan Lamers 1, Ahmed Nabil Belbachir 1, Thomas Bäck 2, Niki van Stein 2 Abstract Catastrophic forgetting can be trivially alleviated by keeping all data from previous tasks in memory. Therefore, minimizing the memory footprint while maximizing the amount of relevant information is crucial to the challenge of continual learning . This paper aims to decrease required memory for memory-based continuous learning algorithms . We explore the options of extracting a minimal amount of information, while maximally alleviating forgetting. We propose the usage of lightweight generators based on Singular Value Decomposition to enhance existing continual learning methods, such as A-GEM and Experience Replay. These generators need a minimal amount of memory while being maximally effective. They require no training time, just a single linear-time fitting step, and can capture a distribution effectively from a small number of data samples. Depending on the dataset and network architecture, our results show a significant increase in average accuracy compared to the original methods. Our method shows great potential in minimizing the memory footprint of memory-based continual learning algorithms. † † tocauthor: Christiaan Lamers, Ahmed Nabil Belbachir, Thomas Bäck and Niki van Stein † † institute: NORCE Norwegian Research Centre, Jon Lilletuns vei 9 H, 3. et, 4879 Grimstad, Norway, and LIACS, Leiden University, Einsteinweg 55, 2333 CC Leiden, The Netherlands † † tocauthor: Christiaan Lamers, Ahmed Nabil Belbachir, Thomas Bäck and Niki van Stein † † institute: NORCE Norwegian Research Centre, Jon Lilletuns vei 9 H, 3. et, 4879 Grimstad, Norway, LIACS, Leiden University, Einsteinweg 55, 2333 CC Leiden, The Netherlands 1 Introduction Continual learning refers to the act of a learner being able to accumulate knowledge over a continual stream of experiences Thrun1998 ; parisi2019continual . Such a learning process allows the learner to refine its abilities over time and integrate skills from different experiences. In practice this means that a learner should be able to learn from a continuous stream of data with a shifting distribution. One of the main challenges in continual learning is overcoming catastrophic forgetting . This refers to the declining performance on previous tasks when multiple tasks are presented to the learner in sequence. Van de Ven et al. van2019three mention three main scenarios in continual learning: task-incremental learning (Task-IL), domain incremental learning (Domain-IL) and class incremental learning (Class-IL). In both Task-IL and Domain-IL, every task has the same class labels, where in Class-IL, the class labels can change at every task. Task-IL and Domain-IL differ in the availability of task labels during test time. In Task-IL, task labels are available during test time, while in Domain-IL, they are not. Of these three scenarios, Task-IL and Domain-IL are most similar, and are the best fit for the methods we would like to propose. We consider Domain-IL to be the hardest challenge and thus the most interesting, therefore our experiments focus on Domain-IL. The problem of catastrophic forgetting can be trivially solved by keeping all data from previous tasks in memory and simply training a neural network on this entire corpus of data. In practice, this is not feasible; thus selecting what to store and what not to store is a core challenge in memory-based continual learning algorithms. This challenge can be extended to the aim of using as little memory as possible, while maximizing the alleviation of catastrophic forgetting. Besides this, Moore’s Law is ending theis2017end , so efficiency both in memory and computational cost is required to advance the state-of-the-art. Our aim is to minimize the amount of required memory by extracting only information on previous tasks that maximally alleviates forgetting. Main contributions: A Singular Value Decomposition (SVD) golub1971singular based method is developed that minimizes the footprint of continual learning algorithms. The benefits have been evaluated of storing abstractions of the data rather than storing the raw data for continual learning tasks. The paper is organized as follows: Section 2 contains a literature review , where the five trends in continual learning are briefly discussed, with a focus on rehearsal-based approaches and the use of generators . Section 3 explains the methodology . It elaborates on Domain Incremental Learning , the mechanisms of the proposed Lightweight Generators and how they are incorporated in rehearsal-based algorithms in a memory efficient way. Section 4 demonstrates the effectiveness of the proposed method through various experimental results . Section 5 contains the conclusion and future work . 2 Literature Review Parisi et al. parisi2019continual describe three main trends in the alleviation of catastrophic forgetting: regularization, rehearsal and architectural strategies. Wang et al wang2024comprehensive extend this list with two more trends: optimization-based and representation-based strategies. 2.1 Five Trends Regularization focuses on restricting changes to weight values in such a way that older tasks are not forgotten. Rehearsal alleviates forgetting by mixing in samples of previous tasks with the current task. Architectural strategies change the network’s architecture during training time, freezing certain parts in order to not forget and adding new nodes in order to keep the ability to learn. The optimization-based approach focuses on altering the optimization algorithm to negate catastrophic forgetting. The representation-based approach focuses on representing the essence of the data in a efficient way through (semi-)supervised learning or meta learning. Rao et al. rao2019continual represents tasks as clusters of datapoints, together with inter- and intra-cluster variation. 2.2 Rehearsal In the rehearsal approach, samples from previous tasks need to be kept in memory. Chaudry et al. chaudhry2019tiny describe multiple strategies to accommodate this. Reservoir sampling decides on adding samples to the memory by random chance. The ring buffer strategy relies on the First In First Out principle. The K-Means strategy decides on sample addition by measuring distances from centroids. The Mean of Features strategy acts in a similar fashion, but measures distances from the mode in feature space. Lamers et al. Lamers_2023_ICCV explore the possibilities of applying K-means on the input space as well as the weight space to select suitable samples, with the added challenge to do so in a task-agnostic setting. 2.3 Generators Another approach for the rehearsal methods is to generate synthetic samples. A multitude of methods exist that fall in this category named generative replay shin2017continual ; van2018generative ; stoianov2022hippocampal ; lesort2019generative . These methods use a generator to synthesize data from previous tasks, which is then mixed with data from the current task and presented to the learner. Van de Ven et al. van2020brain train a variational autoencoder that is used to generate samples. While such a generator is well fitted to capture non-linear latent spaces, the amount of required memory is not as minimal as our approach. Another downside of variational autoencoders is that they need to be trained on a substantial amount of data, which requires a significant amount of time and resources due to the iterative application of back propagation. Nagata et al. mention that the use of generators makes the learner more robust against overfitting versus storing raw samples nagata2024reducing . Alternatively to storing data that a learner would receive as input, it is also possible to store vectors in the weight space of the network. Orthogonal Gradient Descent (OGD) pmlr-v108-farajtabar20a stores gradients of outputs of a neural network and projects gradient

Read the complete text