EFC++: Elastic Feature Consolidation with Prototype Re-balancing for Cold Start Exemplar-free Incremental Learning

Simone Magistri, Tomaso Trinci, Albin Soutif–Cormerais, Joost van de Weijer, Andrew D. Bagdanov · 2025

Read EFC++: Elastic Feature Consolidation with Prototype Re-balancing for Cold Start Exemplar-free Incremental Learning by Simone Magistri, Tomaso Trinci, Albin Soutif–Cormerais, Joost van de Weijer, Andrew D. Bagdanov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10439.txt.

Introduction

Simone Magistri and co-authors tackle machine learning that must add new classes over time, in a 2025 paper. Their setting is harsh: no old examples may be kept, and the first task offers too little data to train a strong base model. In that cold start case, the model's inner traits drift as new tasks arrive.

Their method, called EFC++, slows drift in the directions that mattered most for past tasks. It keeps class averages, called prototypes, and rebalances them after training to fix drift. A matrix built from past traits defines which directions count as important. Tests on four image sets showed a good balance of keeping old skill and learning new classes, ahead of recent rivals.

Catalog record

CREATORSimone Magistri, Tomaso Trinci, Albin Soutif–Cormerais, Joost van de Weijer, Andrew D. Bagdanov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE114 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10439

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EFC++: Elastic Feature Consolidation with Prototype Re-balancing for Cold Start Exemplar-free Incremental Learning Simone Magistri, Tomaso Trinci, Albin Soutif–Cormerais, Joost van de Weijer, Andrew D. Bagdanov Abstract Exemplar-Free Class Incremental Learning (EFCIL) aims to learn from a sequence of tasks without having access to previous task data. In this paper, we consider the challenging Cold Start scenario in which insufficient data is available in the first task to learn a high-quality backbone. This is especially challenging for EFCIL since it requires high plasticity, resulting in feature drift which is difficult to compensate for in the exemplar-free setting. To address this problem, we propose an effective approach to consolidate feature representations by regularizing drift in directions highly relevant to previous tasks and employs prototypes to reduce task-recency bias. Our approach, which we call Elastic Feature Consolidation++ (EFC++) exploits a tractable second-order approximation of feature drift based on a proposed Empirical Feature Matrix (EFM). The EFM induces a pseudo-metric in feature space which we use to regularize feature drift in important directions and to update Gaussian prototypes. In addition, we introduce a post-training prototype re-balancing phase that updates classifiers to compensate for feature drift. Experimental results on CIFAR-100, Tiny-ImageNet, ImageNet-Subset, ImageNet-1K and DomainNet demonstrate that EFC++ is better able to learn new tasks by maintaining model plasticity and significantly outperform the state-of-the-art. † I Introduction In recent years, Deep Neural Networks (DNNs) like Convolutional Neural Networks [ 1 ] and Vision Transformers [ 2 ] have achieved remarkable success for a large variety of artificial intelligence applications but have also introduced additional training complexities. The standard supervised learning paradigm requires massive datasets for robust generalization [ 3 ] , and collecting large amounts of labeled data in a single session can be impractical due to time and cost constraints. Moreover, in real-world applications like autonomous driving [ 4 ] , multimedia forensics [ 5 ] , healthcare [ 6 ] or remote-sensing [ 7 ] , data are not stationary, often necessitating the adaptation of DNNs to new data and tasks. Updating models with new data is challenging due to the phenomenon known as catastrophic forgetting [ 8 , 9 ] : unlike the human brain, when a DNN learns new tasks it quickly forgets how to perform the previously-learned ones. This phenomenon is attributed to the high plasticity of neural networks when learning new tasks, which invariably updates parameters crucial to previous tasks in order to accommodate the new one [ 10 ] . The objective of Class Incremental Learning (CIL) is to integrate new classification tasks into existing models as they arise, without needing to reuse all data from previously seen classes, thereby striking a better balance between performance and training efficiency [ 11 ] . To achieve this, CIL focuses on enhancing the stability of network learning to preserve knowledge from previously seen classes and reduce the risk of catastrophic forgetting. However, increasing stability alone is insufficient, as an overly stable network may struggle to adapt to new classes, thereby inhibiting plasticity. Consequently, the key challenge in CIL is to find the right equilibrium between stability and plasticity during network training. Striking this balance is often referred to as the stability-plasticity dilemma [ 12 , 13 ] . Relying solely on data from the current task to mitigate catastrophic forgetting is challenging because the final deep representation and classifier tend to be highly biased towards the most recent task (a phenomenon known as task-recency bias ). Additionally, the final classifier is never trained to distinguish among all the encountered classes (a phenomenon called inter-task confusion [ 14 ] ). To address this, many recent CIL approaches, known as Exemplar-based Class Incremental Learning (EBCIL) methods, store samples from previous tasks (e.g., 20 per class) as exemplars and replay them while learning new tasks [ 15 , 16 , 17 ] . While effective, these methods require additional storage and computational resources, which can scale with the number of tasks. Furthermore, storing samples can pose significant privacy issues, as it may not always be feasible to retain such data due to privacy concerns. For these reasons, in this paper we focus on the more general and challenging incremental learning setting without exemplars, a paradigm known as Exemplar-free Class Incremental Learning (EFCIL). Elastic Weight Consolidation (EWC) is a pioneering work in EFCIL [ 10 ] . EWC addresses catastrophic forgetting via weight regularization. EWC is an elegant approach based on a Laplace approximation of the previous task posterior. When training on a new task, an approximate Fisher Information Matrix is used to regularize parameter drift to prevent the optimization from harming parameters important for previous tasks, while simultaneously increasing plasticity to learn new tasks. Since computing the full Fisher Information Matrix is intractable (its size is quadratic in the number of network parameters), EWC uses a diagonal approximation of the matrix. This approximation is an aggressive simplification compared to the full matrix and results in a significant performance drop [ 18 , 19 ] . Given the difficulties in regularizing network weights, recent EFCIL methods rely on the more tractable functional regularization paradigm which mitigates network activation drift during incremental learning. Feature distillation, consisting in regularizing feature space drift via \ell_{2} regularization, is a central component in recent state-of-the-art EFCIL approaches [ 20 , 21 , 22 , 23 , 24 ] . Feature distillation is often combined with class prototypes, either learned [ 23 ] or based on class means [ 21 ] , to perform pseudo-rehearsal of feature from previous tasks in order to reduce task-recency bias and inter-task confusion phenomenon. Prototypes – differently from exemplars – offer an efficient and privacy-preserving way to mitigate forgetting. While feature distillation effectively mitigates forgetting through increased stability, it does so at the cost of reduced plasticity for learning new tasks [ 25 ] . A similar emphasis on stability is also observed in other recent EFCIL methods that freeze the backbone after the first task and focus solely on incrementally learning the classifier [ 26 , 27 ] . This emphasis on stability largely stems from the standard evaluation setting for EFCIL methods [ 22 , 21 , 23 , 26 , 27 ] . In this setting, called the Warm Start scenario, the first task includes a large fraction of the total classes, typically 40% or 50%, making stability more critical than plasticity. Since a high-quality feature extractor is learned in the first task, it improves generalization in subsequent tasks. However, when the initial task is smaller, an incremental learning method must increase plasticity to compensate for the lack of a strong initial feature extractor, requiring substantial updates to the backbone in order to effectively learn new tasks. In Figure 1 we give an empirical analysis via linear probing to demonstrate the importance of plasticity in EFCIL for different initial task sizes. These results show that for smaller initial task sizes (e.g. \mathcal{C}_{0}=10 ), the plasticity potential \Delta – defined as the maximum performance gain which can be obtained with a plastic versus a frozen backbone – is highest. This reflects the greater need for feature adaptation in later incremental learning stages, as a weaker initial backbone demands more flexibility to

Read the complete text