Towards Experience Replay for Class-Incremental Learning in Fully-Binary Networks

Yanis BASSO-BERT, Anca MOLNOS, Romain LEMAIRE, William GUICQUERO, Antoine DUPRET · 2025

Read Towards Experience Replay for Class-Incremental Learning in Fully-Binary Networks by Yanis BASSO-BERT, Anca MOLNOS, Romain LEMAIRE, William GUICQUERO, Antoine DUPRET (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07107.txt.

Introduction

Yanis Basso-Bert, Anca Molnos, Romain Lemaire, William Guicquero, and Antoine Dupret wrote a 2025 paper on teaching tiny neural networks new classes over time. Their networks use only binary values, plus or minus one, which suits chips that sip power. Such a network must learn new categories without forgetting old ones, a task called class-incremental learning.

Most binary networks still keep a few full-precision layers. The team built a fully binary design and training method suited to staged learning. They add a loss-balancing trick that weighs old and new classes, and a pre-training step that uses unlabeled data to build transferable features. Work like this serves cameras and sensors that must adapt in the field without a data center.

Catalog record

CREATORYanis BASSO-BERT, Anca MOLNOS, Romain LEMAIRE, William GUICQUERO, Antoine DUPRET
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-07107

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Experience Replay for Class-Incremental Learning in Fully-Binary Networks Yanis BASSO-BERT, Anca MOLNOS, Romain LEMAIRE, William GUICQUERO, Antoine DUPRET Abstract Binary Neural Networks (BNNs) are a promising approach to enable Artificial Neural Network (ANN) implementation on ultra-low power edge devices. Such devices may compute data in highly dynamic environments, in which the classes targeted for inference can evolve or even novel classes may arise, requiring continual learning. Class Incremental Learning (CIL) is a common type of continual learning for classification problems, that has been scarcely addressed in the context of BNNs. Furthermore, most of existing BNNs models are not fully binary, as they require several real-valued network layers, at the input, the output, and for batch normalization. This paper goes a step further, enabling class incremental learning in Fully-Binarized NNs (FBNNs) through four main contributions. We firstly revisit the FBNN design and its training procedure that is suitable to CIL. Secondly, we explore loss balancing, a method to trade-off the performance of past and current classes. Thirdly, we propose a semi-supervised method to pre-train the feature extractor of the FBNN for transferable representations. Fourthly, two conventional CIL methods, i . e ., Latent and Native replay, are thoroughly compared. These contributions are exemplified first on the CIFAR100 dataset, before being scaled up to address the CORE50 continual learning benchmark. The final results based on our 3Mb FBNN on CORE50 exhibit at par and better performance than conventional real-valued larger NN models. A Preprint Keywords Continual Learning \cdot Class-Incremental Learning \cdot Replay Methods \cdot Binary Neural Network \cdot Quantization 1 Introduction Binary Neural Networks (BNNs) [ 1 ] have emerged as a promising avenue for enabling Artificial Neural Network (ANN) deployment on ultra-low power devices, e . g ., always-on processing for computer vision [ 2 , 3 , 4 , 5 , 6 ] . BNN hardware implementations are highly memory- and energy-efficient with respect to conventional hardware realizations (e.g., floating point or 8-bit integer). Furthermore, BNNs are also compatible with a wide variety of emerging hardware technologies, from custom compute-in-memory to emerging optical processing [ 7 , 8 , 9 , 10 , 11 ] . Addressing both memory and computational constraints makes BNNs ideal for edge-based inference tasks. Recently, significant efforts have been made to enhance BNN classification performance through improvements in model structure and training protocols [ 12 , 13 ] . These advancements have been built upon the assumption that the training data is identically distributed to the data encountered during inference. This assumption falls short in dynamic scenarios, in which the concepts ( e . g ., classes) targeted for inference ( e . g ., classification) can evolve or even novel concepts may arise. Such systems necessitate continual learning [ 14 , 15 ] , i . e ., the capability to adapt the model parameters to new concepts without losing the knowledge of old concepts [ 16 ] . The Class Incremental Learning (CIL) methods [ 17 ] [ 18 ] address this issue of learning new knowledge while minimizing forgetting in classification problems. A relatively simple and effective method for CIL is Experience Replay (ER) that stores a subset of past experiences in a buffer. Whenever the data of a new class (or set of classes, i . e ., task) is available, ER retrains the network on a mix of stored data, interleaved with the new-coming data. ER allows the model to learn from both new and old data, thus mitigating forgetting. Recently, the literature starts to address CIL on BNNs [ 19 , 20 ] . Nevertheless, to achieve sufficient performance, these approaches still require several real-valued network layers. To the best of our knowledge, this paper presents the first in-depth study for CIL in Fully Binarized Neural Networks (FBNNs). The main advantage of FBNNs is that their hardware mapping is very efficient. All computations (binarized scalar products with binary inputs) can be performed by highly compact hardware structures, combining point-wise multiplication via logical gates and accumulation using pop-counters [ 21 , 22 ] . To this end, we made four key contributions, each exemplified in part on the CIFAR100 dataset. Finally, the overall approach is validated on the CORE50 dataset [ 23 ] . Firstly, this paper revisits existing work in [ 24 ] , thoroughly details the salient points to guide FBNN design for CIL, and extend previous results with new metrics and a comparison with state-of-the-art. Most BNNs from the literature are actually not fully binary, with their first and last layers in richer, real-value or integer, representations [ 25 ] . Furthermore, to ease convergence, existing BNNs may include Batch Normalization (BN) layers, typically in floating point. To avoid channel-wise BN, we propose to train the network in a single-stage, relying on layer-wise scaling factors only as normalization. The values of the scaling factors are set not from the BN layer parameters, but utilising the dimensions of network topology itself. Furthermore, for the network’s bottleneck we propose a trainable global average pooling to reduce dimensionality of the latent space, making the network robust to spatial translations of the input, despite the low expressivity of binary values. Last, to avoid real-valued input data, we transform the input image pixels into a luminance/chrominance domain, which is further converted into a thermometer encoding [ 26 ] . Our FBNN is compared with the two closest NN designs, without BN [ 27 , 22 ] or claiming only binary arithmetic [ 22 ] . The performance of FBNNs of three model sizes (parameters’ memory footprint) is reported: a 3 Mega bit (Mb), a 9Mb, and a 24Mb. Experiments indicate that our network reaches state-of-the-art performance in an offline setting, i . e ., when all the dataset is available. Furthermore, for the CIL experiments, the contributions in this paper mainly focus on our tiny 3Mb-BNN model. We implement two CIL types of ER, Native replay (storing raw inputs in the buffer) and Latent replay (storing latent features). Secondly, we address the main CIL trade-off, namely adaptability vs retention, i.e., performance on past tasks vs. on the current task. Loss balancing, a possible strategy to control adaptability and retention, is less addressed in the CIL literature. Along with loss balancing, here we explore three loss functions suitable to BNNs, namely, the conventional Categorical Cross-Entropy (CCE), the squared-hinge, potentially better for BNNs [ 28 ] , and a focal loss. We find that a CCE loss suits well CIL in FBNN, for both Latent and Native replay. Balancing the loss significantly enhances the final performance of all tasks, while slightly decreasing adaptation. The proposed FBNN is split in a Feature Extractor (FE) part and a classifier part, as any conventional NN in image classification problems. The common practice for learning feature representation is to perform supervised learning on a pre-training dataset [ 20 ] . Extracted features can be discriminative for pre-training classes, however, there is no guarantee that they are also discriminative for later tasks. It becomes an issue for Latent replay, in which the feature extractor is not updated. Thirdly, the supervised pre-training approach is challenged, proposing a semi-supervised alternative to learn more transferable features in FBNNs. Supervised training is

Read the complete text