CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning
Julien Lefebvre, Stefan Duffner, Mathieu Lefort · 2026
Read CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning by Julien Lefebvre, Stefan Duffner, Mathieu Lefort (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31275.txt.
Introduction
Lefebvre and two colleagues present CLIMB, a memory system for learning from an endless unlabeled image stream. Online continual self-supervised learning must form good representations with no task boundaries and little storage. Older methods either replay stored examples or rely on regularisation alone to stop forgetting.
CLIMB joins both routes with a hierarchical centroid memory that groups similar images under cluster centres within a fixed budget. Those centres supply hard-to-tell-apart examples for contrastive learning and cover the spread of past data, while distillation on replayed items limits representation drift. On the Split CIFAR-100 and Split ImageNet-100 benchmarks, plus a new protocol with uneven task mixes, CLIMB beats leading methods in its class.
Catalog record
| CREATOR | Julien Lefebvre, Stefan Duffner, Mathieu Lefort |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31275
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning Julien Lefebvre, Stefan Duffner, Mathieu Lefort Abstract Online Continual Self-Supervised Learning (OCSSL) aims to learn representations from a continuous stream of unlabeled data, without knowledge of task boundaries and under memory constraints. Existing methods rely either on replay buffers that exploit latent space structure, or on regularization alone. We present CLIMB ( Continual Learning with Intelligent Memory Bank ), which combines both simultaneously. Our method introduces a hierarchical centroid-based memory, bounded in total number of stored images, combined with knowledge distillation on replayed examples to limit representation drift. The memory groups similar images into centroids, providing hard-to-discriminate examples for contrastive learning while covering the diversity of observed distributions. Experiments on Split CIFAR-100 and Split ImageNet-100, on standard benchmarks from the state-of-the-art as well as a new protocol with irregular task distributions show that CLIMB outperforms state-of-the-art OCSSL methods. Source code is available at : https://github.com/lefebvju/climb 1 Introduction Self-Supervised Learning (SSL) has established itself as the dominant paradigm for representation learning from unlabeled data ( 3 ; 11 ; 4 ) . SSL methods exploit the intrinsic structure of data through augmentations. Contrastive approaches ( 3 ) attract different views of the same image while repelling those of different images. Non-contrastive approaches ( 11 ; 4 ) prevent representational collapse through architectural asymmetry without requiring negative examples. However, most SSL methods assume that all training data is jointly available, which is not realistic when data evolves over time. In such settings, these systems either require full retraining from scratch at each update, or suffer from catastrophic forgetting , the tendency of neural networks to overwrite weights associated with past tasks when learning new ones ( 13 ) . The domain of Continual Learning (CL) addresses this challenge by enabling models to acquire new knowledge incrementally without erasing prior representations ( 25 ) . A sub-branch of continual learning focuses on unannotated data, making it possible to exploit far larger data volumes by removing the need for labels. The intersection of CL and SSL defines Continual Self-Supervised Learning (CSSL), an emerging field explored by several recent works ( 8 ; 10 ; 29 ) . Generally in CSSL, methods assume that multiple epochs can be performed on each task and that task boundaries are known, allowing catastrophic forgetting mitigation mechanisms to be triggered explicitly at each transition. We focus on a subdomain of CSSL, Online Continual Self-Supervised Learning (OCSSL) ( 27 ) , where the model performs a single pass over the incoming data stream using mini-batches, with no access to task boundaries. This setting is particularly relevant for applications requiring rapid adaptation and continuous representation evaluation, for example an autonomous agent exploring a novel environment. Fair comparison between methods requires controlling both memory capacity and computational budget simultaneously, a constraint barely addressed in the existing literature. In this paper, we present CLIMB , which addresses the OCSSL setting by combining a novel strictly bounded hierarchical centroid-based memory with a classical EMA-based regularization. Our contributions are: 1. A hierarchical centroid-based memory inspired by PCMC ( 23 ) , compatible with the OCSSL setting under strict memory constraints. Unlike PCMC, CLIMB operates with full-image representations rather than patches, removes both the offline pretraining phase and the sleep phases, and imposes a strict global bound on memory size. Combined with knowledge distillation on replayed examples following CLA ( 5 ) , this memory improves performance in most configurations, with a particularly pronounced advantage when data is observed in a more fragmented fashion, i.e., when each task covers a smaller subset of classes. 2. A new evaluation protocol with irregular task distributions for the OCSSL setting, where the number of classes per task varies randomly, following irregular distributions previously explored in supervised continual learning ( 14 ) , in order to measure the robustness of OCSSL methods against more varied continual learning configurations than the balanced tasks usually considered. The related work is presented in Section 2 , followed by a detailed description of our approach (Section 3 ). Experimental protocols and analysis of results are then developed in Sections 4 and 5 , before concluding and discussing perspectives in Section 6 . 2 Related Work In continual learning, three main families of methods have been developed. Regularization-based methods constrain network weight updates based on their importance to previous tasks. EWC ( 13 ) exploits the Fisher information matrix to estimate the importance of each parameter and penalizes their modification when learning new tasks. SI ( 28 ) extends this idea by evaluating parameter importance along the full learning trajectory. Both methods operate in a supervised setting. A second regularization family targets model outputs rather than parameters, using a frozen previous model as a teacher via a distillation loss. LwF ( 16 ) is the founding work of this approach. CaSSLe ( 8 ) adapts it to the offline self-supervised setting via a dedicated projection head. Architecture-based methods dynamically adapt model structure to allocate capacity to new tasks. Progressive Neural Networks ( 19 ) dedicate a new backbone to each task while adding lateral connections to reuse past representations. U-TELL ( 22 ) extends this paradigm to the unsupervised setting by dedicating a new expert module to each incoming task. Replay-based methods revisit past data during training. They divide into two main categories: generative approaches, which use a generative model to synthesize past examples without requiring explicit storage of input ( 6 ; 22 ) , and episodic memory approaches, which maintain a buffer containing a subset of past data for replay during training ( 17 ; 23 ; 5 ) . These approaches have been developed in both supervised ( 6 ) and self-supervised settings ( 17 ; 22 ; 5 ) . These three families are not mutually exclusive and are frequently combined to leverage their complementarity. In a supervised setting, methods such as iCaRL ( 18 ) combine replay and distillation to better preserve past knowledge. In a self-supervised setting, replayed examples are also frequently combined with knowledge distillation regularization, where a frozen previous model constrains current representations from drifting excessively ( 8 ; 10 ; 5 ) . U-TELL ( 22 ) combines architectural modifications with generative replay in an unsupervised setting. Online supervised continual learning has been extensively studied ( 1 ; 14 ; 2 ) , where the single-pass constraint requires updating representations from each mini-batch only once without storing the full dataset. Online Continual Self-Supervised Learning (OCSSL) extends this challenge to the unlabeled setting, combining the absence of annotations, a single-pass data stream, and unknown task boundaries. Existing methods augment a standard SSL backbone with mechanisms from the families described above: a regularization loss constraining representation drift, an episodic memory for replaying past examples, or a combination of both. Comparing OCSSL methods equitably is non-trivial, as performance depends on both memory capacity and computational budget. A method with access to more stored examples benefits from more diverse replay, while a method performing more gradient updates exploits that diversity more effectively. Without controlling both simultaneously, comparisons become meaningless: a well-organized memory paired with a larger budget would trivially outperform a simpler method under tighter constraints. CLA ( 5 ) identifies