What is the role of memorization in Continual Learning?
Jędrzej Kozal, Jan Wasilewski, Alif Ashrafee, Bartosz Krawczyk, Michał Woźniak · 2025
Read What is the role of memorization in Continual Learning? by Jędrzej Kozal, Jan Wasilewski, Alif Ashrafee, Bartosz Krawczyk, Michał Woźniak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17664.txt.
Introduction
Jędrzej Kozal and co-authors dissect the double-edged role of memorization in continual learning, in a 2025 paper. Continual learning systems must absorb new tasks without erasing old ones. Memorization of examples shapes both learning and forgetting. Prior work tied it mostly to generalization and privacy.
Their experiments score examples by how strongly models memorize them. Highly memorized examples turn out to be forgotten fastest. Yet some memorization is needed for peak performance. At small memory budgets, losing ordinary examples hurts more. The findings split a seemingly single effect into regimes. Practitioners gain guidance on what to store in replay buffers. The paper reframes memorization as a resource to manage, not a flaw.
Catalog record
| CREATOR | Jędrzej Kozal, Jan Wasilewski, Alif Ashrafee, Bartosz Krawczyk, Michał Woźniak |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17664
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What is the role of memorization in Continual Learning? Jędrzej Kozal, Jan Wasilewski, Alif Ashrafee, Bartosz Krawczyk, Michał Woźniak Abstract Memorization impacts the performance of deep learning algorithms. Prior works have studied memorization primarily in the context of generalization and privacy. This work studies the memorization effect on incremental learning scenarios. Forgetting prevention and memorization seem similar. However, one should discuss their differences. We designed extensive experiments to evaluate the impact of memorization on continual learning. We clarified that learning examples with high memorization scores are forgotten faster than regular samples. Our findings also indicated that memorization is necessary to achieve the highest performance. However, at low memory regimes, forgetting regular samples is more important. We showed that the importance of a high-memorization score sample rises with an increase in the buffer size. We introduced a memorization proxy and employed it in the buffer policy problem to showcase how memorization could be used during incremental training. We demonstrated that including samples with a higher proxy memorization score is beneficial when the buffer size is large. 1 Introduction Memorization plays an important role in deep learning [ 18 , 19 ] . Deep learning models are capable of memorizing the entire dataset with completely random training labels [ 57 ] , achieving almost perfect training accuracy. Methods that limit memorization, such as privacy-preserving algorithms [ 43 ] , fail to achieve classification accuracy comparable to that of standard training methods. Motivated by these observations, later works have shown that achieving high test set performance requires memorization [ 18 ] . So far, memorization has been studied primarily through the lens of generalization or privacy [ 53 ] . Continual Learning (CL) [ 15 ] is the area of machine learning that aims to prevent catastrophic forgetting [ 20 ] in the models trained incrementally with data sampled from different data distributions. There were many methods developed that aim to prevent forgetting [ 40 ] . Counterintuitively, memorization and forgetting prevention are not the same. According to the definition of Feldman [ 19 ] , memorization refers to the phenomenon where some sample is classified correctly when it is included in the training data. Once removed from the training dataset, it is no longer properly classified. It means that the model can only memorize the sample label without learning any pattern. Such phenomena could occur when the sample has an incorrect label due to a labeling error or when the sample belongs to a long tail [ 18 ] (meaning it could be a minority class in an imbalanced setting or it could contain weakly represented features in the dataset). The opposite of forgetting is not memorization, but complete knowledge retention [ 15 ] . In this case, we are interested in keeping the predictive performance high on past tasks, regardless of whether the sample belongs to the long tail or not. While forgetting and memorization are different, studying them requires understanding how knowledge is retained in the network. We found that there are relatively few works [ 28 , 38 , 48 ] that study the connection between memorization and forgetting. Moreover, these studies examine forgetting in the context of privacy or label noise, with no focus on the change in data distribution. For this reason, we want to study what is the role of memorization in an incremental training setup, and how Continual Learning is impacted by memorization of training data. Our key findings could be summarized as follows: • Increasing the number of classes in dataset increases memorization. • The higher the memorization score of an example, the greater its susceptibility to forgetting. • When training with full access to data from past tasks, the classification accuracy of memorized samples remains high. • A computationally efficient proxy for memorization score can effectively guide buffer policies to improve the performance of incremental training. • The importance of examples, with a significant proxy memory score, increases for large buffers. 2 Related Works 2.1 Memorization Memorization [ 57 ] in deep learning refers to a phenomenon where deep neural networks learn specific details or particular features of individual training examples, rather than extracting common patterns or generalized features of the underlying data distribution. This can lead to a model’s reliance on rote recall, potentially impacting its ability to generalize to new, unseen data and raising concerns about security and privacy [ 53 ] . In [ 57 ] , it was shown that overparametrized neural networks are capable of fitting training data with random labels with high training accuracy. Further work [ 3 ] studied the impact of architecture and dataset size on memorization. It also shows that general concepts are learned early in training, and memorization could lead to more complicated decision boundaries. Anagnostidis et al. [ 2 ] showed that even with random labels, deep neural networks learn some features that are beneficial for classification on data with original labels. Further works [ 10 , 39 , 47 ] show that memorization does not necessarily lead to overfitting. In [ 18 ] , it was shown that obtaining high accuracy on the test set requires memorization. The atypical samples from the long tail are too few to enable proper representation learning, leading to memorization as the only way of obtaining high performance on the test set. In [ 19 ] Feldman et al. introduced an influence score that allows for estimating the influence of training samples on test samples predictions, finding that there are training examples that impact the correct classification of unseen long-tail data. So far, there is no consensus among researchers on what parts of the neural network are responsible for memorization [ 53 ] . Some evidence [ 2 ] suggested that the last layers are used for memorization. Other studies [ 39 ] indicated that neurons responsible for memorization are scattered across the whole network. In [ 19 ] , it was shown that a classifier is not responsible for memorization, suggesting that memorization is a phenomenon that concerns primarily representation learning. 2.2 Continual Learning Continual learning addresses the challenge of training models on a sequence of tasks with differing data distributions, rather than on a single i.i.d. dataset [ 15 ] . A major issue in this setting is catastrophic forgetting, where performance on earlier tasks sharply deteriorates as the model learns new ones [ 20 ] . To tackle this, continual learning methods are generally grouped into three categories. Regularization-based methods reduce forgetting by constraining changes to important parameters. Elastic Weight Consolidation (EWC) adds a regularization term to penalize updates to critical weights [ 29 ] . Learning without Forgetting (LwF) maintains previous knowledge by using pseudo-labels from earlier task classifiers [ 33 ] . FeTrIL [ 45 ] applies a pseudo-feature generation strategy with the usage of a frozen backbone. Magistri et al. [ 37 ] address feature drift by regularizing direction relevant for past tasks. Rehearsal-based methods rely on memory buffers to replay examples from previous tasks [ 13 ] . Gradient Episodic Memory (GEM) and its efficient variant aGEM, constrain gradient updates to prevent loss on earlier tasks [ 12 , 35 ] . More recent work addresses the limitations of