A Conformal Predictive Measure for Assessing Catastrophic Forgetting
Ioannis Pitsiorlas 2 * , Nour Jamoussi 2 * , Marios Kountouris 2 3 · 2025
Read A Conformal Predictive Measure for Assessing Catastrophic Forgetting by Ioannis Pitsiorlas 2 * , Nour Jamoussi 2 * , Marios Kountouris 2 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10677.txt.
Introduction
Ioannis Pitsiorlas, Nour Jamoussi, and Marios Kountouris brought in a new yardstick in 2025 for a familiar AI failure: forgetting old skills while learning new ones. Continual learning systems study tasks in sequence, and catastrophic forgetting erases earlier competence. Current measures lean on accuracy alone. The authors instead use conformal forecast, a numeric method that wraps forecasts with confidence sets.
Their metric, the Conformal Forecast Confidence Factor, watches how the model's confidence on before learned tasks evolves as new tasks arrive. Self-tuning conformal methods keep the measure useful and dynamic. Across four benchmark data sets, the factor correlated strongly with the actual accuracy retained on old tasks. A trustworthy forgetting gauge helps builders decide when to intervene with rehearsal or regularization. It also makes continual learning claims easier to audit, because confidence actions is harder to fake than a single score.
Catalog record
| CREATOR | Ioannis Pitsiorlas 2 * , Nour Jamoussi 2 * , Marios Kountouris 2 3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-10677
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Conformal Predictive Measure for Assessing Catastrophic Forgetting Ioannis Pitsiorlas 2 * , Nour Jamoussi 2 * , Marios Kountouris 2 3 Abstract This work introduces a novel methodology for assessing catastrophic forgetting (CF) in continual learning. We propose a new conformal prediction (CP)-based metric, termed the Conformal Prediction Confidence Factor (CPCF) , to quantify and evaluate CF effectively. Our framework leverages adaptive CP to estimate forgetting by monitoring the model’s confidence on previously learned tasks. This approach provides a dynamic and practical solution for monitoring and measuring CF of previous tasks as new ones are introduced, offering greater suitability for real-world applications. Experimental results on four benchmark datasets demonstrate a strong correlation between CPCF and the accuracy of previous tasks, validating the reliability and interpretability of the proposed metric. Our results highlight the potential of CPCF as a robust and effective tool for assessing and understanding CF in dynamic learning environments. Fig. 1 : Proposed framework overview for assessing catastrophic forgetting in continual learning via conformal prediction. I Introduction Current machine learning (ML) models are predominantly designed for single-task applications, where they undergo iterative training over multiple epochs on a static dataset until convergence is reached [ 1 ] . However, significant challenges arise when models are trained exclusively on fixed datasets, particularly in real-world scenarios where new tasks or classes may frequently emerge over time. For instance, the transition to 6G networks in wireless communications calls for adaptive resource allocation models that can accommodate new devices or services without requiring complete retraining. As user demands and network configurations change, traditional ML models often fail to generalize effectively, rendering full retraining costly and impractical in dynamic environments. A similar issue arises in healthcare, where ML models used for diagnostics are typically trained on known diseases but must also adapt to new conditions, such as emerging virus strains or rare diseases, without compromising accuracy on previously learned cases [ 1 ] . These challenges highlight the need for ML models that can continuously learn and adapt to dynamic, real-world environments while preserving their performance on previously learned tasks. To address this challenge, continual learning (CL), also known as lifelong or incremental learning, has gained considerable attention. CL aims to incrementally learn from a sequence of tasks or data points while preserving the performance of the model as if all data were available concurrently [ 2 ] . A key requirement of a CL system is its ability to adapt to new tasks or shifts in data distribution without relying on access to previously encountered data [ 3 ] . This capability is particularly critical in scenarios where data privacy is a priority, such as under General Data Protection Regulation (GDPR) compliance, where organizations are obligated to restrict data usage and storage beyond a specified retention period. These restrictions significantly complicate the process of reusing historical datasets for retraining. Moreover, the high computational cost of retraining, especially for large-scale models, presents another major barrier when integrating new data [ 4 ] . By addressing both data privacy and computational efficiency, CL offers a promising solution for enabling models to evolve over time while remaining compliant with privacy regulations and minimizing resource demands. Continual learning systems often suffer from catastrophic forgetting (CF), a phenomenon in which previously acquired knowledge is overwritten as new information is learned [ 5 ] . This typically occurs when a model adapts to a shift in the data distribution, resulting in a degradation in performance on earlier tasks or previously encountered distributions. Distribution shifts, such as covariate, label, and concept shifts, refer to changes in the underlying data distribution encountered by a model during training or deployment, often leading to significant performance degradation. Covariate shift involves changes in the input distribution, while the relationship between the input and output remains stable [ 6 ] . Label shift refers to changes in the distribution of output labels while the input distribution stays the same [ 7 ] . Concept shift refers to changes in the input-output relationship itself, which often arises when new tasks are introduced [ 8 ] . Over the years, numerous methods have been proposed in CL to address the challenge of CF. These approaches generally include regularization-based approaches, such as Elastic Weight Consolidation (EWC) [ 9 ] , Online Structured Laplace Approximations [ 10 ] , replay-based approaches, such as Reservoir Sampling [ 11 ] , and optimization-based strategies, such as Gradient Episodic Memory, among others [ 12 ] . A significant limitation of current CL methods is their tendency to overlook predictive uncertainty, often leading to overconfident yet unreliable decisions [ 13 ] . This limitation becomes particularly pronounced and problematic in real-world applications, where models encounter dynamically evolving tasks and data distributions. In high-stakes domains such as healthcare (e.g., risk of misdiagnosis), autonomous systems, or intrusion detection, overconfidence in predictions can result in critical errors, compromised safety, and severe real-world consequences [ 14 , 15 ] . Quantifying predictive confidence is essential not only to improve decision-making, but also to effectively assess and mitigate CF. Traditional methods, such as softmax-based distillation, often produce overconfident predictions and fail to capture the underlying uncertainty in model parameters or data distributions [ 16 ] . In contrast, uncertainty-aware approaches, including those leveraging Bayesian techniques or ensemble methods, have shown promising potential. However, these methods are typically computationally intensive and difficult to scale in CL settings, where efficiency and adaptability are critical requirements [ 16 ] . Conformal prediction (CP) has been successfully applied in various domains, including medical diagnostics, anomaly detection, and uncertainty quantification, to provide calibrated confidence measures for model predictions [ 17 ] . As a model-agnostic and principled framework, CP offers an effective way to quantify uncertainty by providing calibrated confidence levels that remain reliable and can adapt to dynamic learning environments. In this work, we propose a conformal predictive measure for assessing CF, termed the Conformal Prediction Confidence Factor (CPCF). By integrating CP into the CL pipeline, our approach enables an online evaluation framework for CF, enhancing the evaluation process to better reflect the challenges of dynamic real-world environments. Our empirical results demonstrate a strong correlation between the proposed CPCF and model accuracy, in two different learning settings and four benchmark datasets, underscoring its effectiveness in detecting CF. II Proposed Methodology In this section, we introduce Adaptive CP [ 17 ] for classification tasks and propose a new metric, the CPCF, to assess CF in CL settings. An overview of the proposed framework is shown in Figure 1 . II-A Data Splitting To implement CP, the training dataset is split into two subsets according to a predefined calibration ratio. • Effective Training Data: The portion of the dataset that is used to train the model. • Calibration Data: The remaining portion of the dataset, specified by the calibration ratio, is reserved for computing conformal scores, which are essential for forming prediction sets during inference. For example, if the calibration ratio is set to 0.1 , then 90% of the data is allocated for training while the remaining 10% is used for calibration. Clearly, the calibration ratio influences the interplay between training and calibration, balancing predictive performance against the reliability of uncertainty