Class Incremental Learning with Task-Specific Batch Normalization and Out-of-Distribution Detection
Zhiping Zhou, Xuchen Xie, Yiqiao Qiu, Run Lin, Wei-Shi Zheng, Ruixuan Wang · 2024
Read Class Incremental Learning with Task-Specific Batch Normalization and Out-of-Distribution Detection by Zhiping Zhou, Xuchen Xie, Yiqiao Qiu, Run Lin, Wei-Shi Zheng, Ruixuan Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00430.txt.
Description
Abstract This study focuses on incremental learning for image classification, exploring how to reduce catastrophic forgetting of all learned knowledge when access to old data is restricted. The challenge lies in balancing plasticity (learning new knowledge) and stability (retaining old knowledge).
Catalog record
| CREATOR | Zhiping Zhou, Xuchen Xie, Yiqiao Qiu, Run Lin, Wei-Shi Zheng, Ruixuan Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00430
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Class Incremental Learning with Task-Specific Batch Normalization and Out-of-Distribution Detection Zhiping Zhou, Xuchen Xie, Yiqiao Qiu, Run Lin, Wei-Shi Zheng, Ruixuan Wang Abstract This study focuses on incremental learning for image classification, exploring how to reduce catastrophic forgetting of all learned knowledge when access to old data is restricted. The challenge lies in balancing plasticity (learning new knowledge) and stability (retaining old knowledge). Based on whether the task identifier (task-ID) is available during testing, incremental learning is divided into task incremental learning (TIL) and class incremental learning (CIL). The TIL paradigm often uses multiple classifier heads, selecting the corresponding head based on the task-ID. Since the CIL paradigm cannot access task-ID, methods originally developed for TIL require explicit task-ID prediction to bridge this gap and enable their adaptation to the CIL paradigm. In this study, a novel continual learning framework extends the TIL method for CIL by introducing out-of-distribution detection for task-ID prediction. Our framework utilizes task-specific Batch Normalization (BN) and task-specific classification heads to effectively adjust feature map distributions for each task, enhancing plasticity. With far fewer parameters than convolutional kernels, task-specific BN helps minimize parameter growth, preserving stability. Based on multiple task-specific classification heads, we introduce an “unknow” class for each head. During training, data from other tasks are mapped to this unknown class. During inference, the task-ID is predicted by selecting the classification head with the lowest probability assigned to the unknown class. Our method achieves state-of-the-art performance on two medical image datasets and two natural image datasets. The source code is available at https://github.com/z1968357787/mbn_ood_git_main . Journal: Neurocomputing 1 Introduction Deep learning models have been extensively applied across various domains, including image recognition [ 77 , 45 ] , natural language processing [ 13 , 36 ] , generative adversarial networks [ 67 , 66 ] , and industrial applications such as multimodal arc detection [ 73 , 74 ] . However, they usually rely on large amounts of data for training to achieve high performance. In real-world scenarios, it is often difficult to obtain data for all necessary classes at once, leading to data arriving in multiple phases. Moreover, the classes appearing in different phases are typically non-overlapping. Consequently, AI models often need to learn new classes gradually. With privacy and memory constraints, models can only access data from the current phase and are restricted to retrieve data from previous phases. Research has found that under these circumstances, models are susceptible to catastrophic forgetting [ 17 ] , a phenomenon characterized by a substantial decline in performance on previously learned tasks. Therefore, enabling models to continually learn new knowledge without forgetting old knowledge is one key challenge for promoting the widespread application of deep learning based AI across various fields. Multiple approaches have been proposed to alleviate catastrophic forgetting. However, they often struggle to balance the model’s stability (ability to retain old knowledge) and plasticity (ability to learn new knowledge), and may also face challenges in controlling parameter growth. For instance, methods based on model parameter regularization [ 34 , 37 , 70 ] aim to protect crucial neural network weights from being updated drastically through constraints, thus preventing old knowledge loss. However, this protection can lead to model rigidity, making it difficult for the model to assimilate new knowledge. Another type of method is the dynamic expansion of model structure [ 50 , 15 , 75 , 80 ] , where new layers or sub-modules are added for new tasks. Although such methods significantly enhance model plasticity, they inevitably increase memory demands. Therefore, effectively adding network components without substantially increasing the memory burden is a fundamental issue for this type of method. Overall, existing methods struggle to achieve a good balance between plasticity, stability and memory control, which is the core challenge in incremental learning. In a regular sequential multi-task learning process (without incremental learning), a single feature extractor without any restriction is used to fit the distinctive knowledge representations of each task sequentially. During this process, the feature extractor focuses on learning new tasks only, leading to catastrophic forgetting of old tasks. Minimizing updates of the feature extractor can substantially mitigate or completely avoid catastrophic forgetting. In this paper, we propose a method that minimizes changes to learned feature representations by introducing task-specific Batch Normalization (BN) layers and classification heads. Since BN layers contain far fewer parameters than convolutional layers in typical CNNs, they can be added for each task with minimal parameter growth. These task-specific modules are trained and then preserved in subsequent continual learning, which helps mitigate catastrophic forgetting and maintain model stability. Moreover, BN effectively reshapes feature distributions, enhancing the model’s ability to capture task-specific characteristics and improving learning plasticity. Because the task identifier (task-ID) is unavailable in the class-incremental learning (CIL) setting, an additional mechanism is required at inference time to select the appropriate task-specific modules. For each task-specific BN layer and classification head, data from other tasks can be regarded as out-of-distribution (OOD) samples. To leverage this property, we augment each classification head with an “unknown” class. During training, samples from other tasks are mapped to this class, enabling the classification heads to perform implicit OOD detection. During inference, the task-ID is predicted by selecting the head with the lowest probability assigned to the unknown class, allowing the model to choose the most appropriate task-specific modules. This work extends our prior conference publication [ 71 ] where we made two foundational contributions: 1. Pioneered task-specific BN in Task-Incremental Learning (TIL); 2. Established theoretical guarantees. Under task-ID availability, we have demonstrated that task-specific BN layers and classifier heads can significantly enhance within-task classification performance. Building upon these foundations, this study includes the following major contributions: 1. We introduce task-specific BN in the CIL paradigm for the first time. 2. Based on multiple task-specific classifier heads, we introduce out-of-distribution detection for task-ID prediction, demonstrating that a task-ID prediction mechanism can effectively extend TIL to the CIL setting, where the task-ID is unknown. 3. Our proposed method achieves state-of-the-art CIL performance on two medical image datasets and two natural image datasets, achieving a better balance among model stability, plasticity, and parameter growth. 2 Related Work 2.1 General Overview Incremental learning for image classification is divided into two main paradigms, namely, task incremental learning [ 25 , 60 ] (TIL) and class incremental learning [ 25 , 60 ] (CIL). Both TIL and CIL learn new class knowledge sequentially in a task-based manner during training. Each task contains multiple classes, and the class sets across tasks are mutually disjoint. During testing, TIL provides the task identity for each sample, commonly referred to as the task identifier (task-ID), whereas CIL does not provide this information. The CIL methods can be transferred to TIL by focusing only on specific categories associated with the task-ID during testing. However, adapting methods from TIL to CIL remains a significant challenge, mainly because choosing the correct classifier head from multiple options to make the final prediction becomes particularly challenging without task-ID. Thus, for most existing methods, the main difference between these two paradigms lies in that the TIL paradigm has multiple classifier heads specific to certain tasks, while the CIL paradigm typically has only one classifier head that adapts