Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training
Brown Ebouky [email protected], Ajad Chhatkuli, Cristiano Malossi, Christoph Studer, Roy Assaf, Andrea Bartezzaghi · 2025
Read Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training by Brown Ebouky [email protected], Ajad Chhatkuli, Cristiano Malossi, Christoph Studer, Roy Assaf, Andrea Bartezzaghi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17816.txt.
Introduction
Brown Ebouky and 5 co-authors ask how a vision model can keep learning when it moves to a new field with little data. Their 2025 paper deals with self-taught pre-training, where a model learns from images with no tags. The target task is to mark each part of an image by kind.
Their method, GLARE, adds patch-level tweaks and a rule that keeps nearby areas alike in sense. Just small add-on parts of the model are trained, while the main body stays fixed. Tests on more than one marking bench show gains for the later task. The plan keeps cost low while the model adapts.
Catalog record
| CREATOR | Brown Ebouky [email protected], Ajad Chhatkuli, Cristiano Malossi, Christoph Studer, Roy Assaf, Andrea Bartezzaghi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17816
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training Brown Ebouky [email protected], Ajad Chhatkuli, Cristiano Malossi, Christoph Studer, Roy Assaf, Andrea Bartezzaghi Abstract Self-supervised learning (SSL) has emerged as a central paradigm for training foundation models by leveraging large-scale unlabeled datasets, often producing representations with strong generalization capabilities. These models are typically pre-trained on general-purpose datasets such as ImageNet and subsequently adapted to various downstream tasks through finetuning. While recent advances have explored parameter-efficient strategies for adapting pre-trained models, extending SSL pre-training itself to new domains—particularly under limited data regimes and for dense prediction tasks—remains underexplored. In this work, we address the problem of adapting vision foundation models to new domains in an unsupervised and data-efficient manner, specifically targeting downstream semantic segmentation. We propose GLARE (Global Local and Regional Enforcement), a novel continual self-supervised pre-training task designed to enhance downstream segmentation performance. GLARE introduces patch-level augmentations to encourage local consistency and incorporates a regional consistency constraint that leverages spatial semantics in the data. For efficient continual pre-training, we initialize Vision Transformers (ViTs) with weights from existing SSL models and update only lightweight adapter modules—specifically UniAdapter—while keeping the rest of the backbone frozen. Experiments across multiple semantic segmentation benchmarks on different domains demonstrate that GLARE consistently improves downstream performance with minimal computational and parameter overhead. 1 Introduction Self-supervised learning (SSL) has revolutionized the training of foundation models by enabling the extraction of rich, generalizable features from vast unlabeled datasets ( Caron et al. 2021 ; Chen et al. 2021 ; Oquab et al. 2024 ; Assran et al. 2023 ) . This paradigm has proven to be particularly valuable in computer vision, where the abundance of unlabeled images can be leveraged to learn robust visual representations without the need for expensive manual annotations. Recent advances in SSL have introduced sophisticated techniques, spanning from innovative data augmentation strategies ( Grill et al. 2020 ) and masked image modeling ( He et al. 2022 ) to enhanced feature matching through self-attention mechanisms ( Su et al. 2023 ; Su & Ji 2024 ) and refined loss functions. While these developments have led to powerful models pre-trained on extensive generic datasets like ImageNet ( Deng et al. 2009 ) , they often fall short when confronted with specialized technical domains. This limitation becomes particularly apparent in real-world scenarios where domain-specific data is scarce, especially labeled data. The challenge is further compounded when considering dense prediction tasks like semantic segmentation, which demand more fine-grained semantic understanding compared to classification tasks. These issues have therefore sparked significant interest in adaptation techniques, particularly finetuning and continual learning to bridge the gap between generic pre-training and domain-specific applications. Our research focuses on enhancing foundation model features for task-specific limited data scenarios through continual pre-training within a pure SSL framework. While previous research has explored this direction by training batch normalization layers using conventional SSL approaches for classification tasks ( Reed et al. 2022 ) , these studies do not fully investigate other aspects of the SSL paradigm, such as augmentations and matching, in limited data scenarios. Furthermore, while extensive research exists on continual pre-training for classification datasets ( Lin et al. 2022 ; Cheng et al. 2023 ; Reed et al. 2022 ; Tang et al. 2024 ) , the distinct challenges posed by semantic segmentation warrant separate investigation, as features of foundation models often perform differently between classification and segmentation tasks ( Su & Ji 2024 ; Caron et al. 2021 ; Oquab et al. 2024 ) . Therefore, we focus our work on identifying an appropriate data-efficient technique for the continual SSL pre-training of foundation models with limited unlabeled data, tailored for downstream semantic segmentation. Our key contributions include: • We explore the benefits of SSL-based continual pre-training for semantic segmentation under limited data conditions. To this aim, we employ a trainable adapter ( Lu et al. 2023 ) after the self-attention layers in ViTs, in order to mitigate catastrophic forgetting during continual SSL pre-training. • We propose a data-efficient augmentation strategy centered on patch-wise strong blurring, instead of the typical patch-wise or block-wise masking approaches. This seemingly simple modification proves particularly effective in facilitating the learning of inter-patch relationships, especially in limited data settings. • We propose explicit local and regional consistency enforcement to improve learning dense features with limited data. These constraints help to learn more distinct spatial features crucial for segmentation tasks. Specifically, we propose a regional consistency mechanism by penalizing feature inconsistency between the student (base encoder) and teacher (momentum encoder) within spatial patch groups, processed through a cross-attention layer. In this context, we sample the region features based on attention maps to leverage the semantic content already learned by the original model. Alongside the local patch-wise consistency, the regional consistency helps in learning better spatially distinct features suitable for semantic segmentation. We provide model-specific experiments with existing state-of-the-art foundation models, while comparing our proposed SSL with standard SSL approaches. We validate our approach through experiments across four datasets: ADE20k ( Zhou et al. 2017 ) , Pascal Context ( Mottaghi et al. 2014 ) , Cityscapes ( Cordts et al. 2016 ) and LoveDA ( Wang et al. 2021a ) (satellite images), showing performance improvements on downstream segmentation task after applying the proposed SSL approach in continual pre-training. 2 Related Works 2.1 SSL for global image understanding SSL methods for vision aim to learn rich visual representations from large-scale unlabeled data by enforcing various learning objectives and paradigms. One prominent paradigm is joint embedding networks, exemplified by methods like MoCo ( He et al. 2020 ) and DINO ( Caron et al. 2021 ) . MoCo employs contrastive learning with a memory bank for gathering negative examples ( He et al. 2020 ; Wu et al. 2018 ) , while later approaches, such as SimCLR ( Chen et al. 2020b ) and MoCo-V3 ( Chen et al. 2021 ) , eliminate the memory bank and use training batch samples as negatives. In contrast, methods like DINO ( Caron et al. 2021 ) , BYOL ( Grill et al. 2020 ) , and SimSiam ( Chen & He 2021 ) forego negative pairs altogether, focusing instead on positive pair similarity. Techniques for achieving this include contrastive learning (e.g., ZeroCL ( Zhang et al. 2022 ) ), clustering (e.g., SwAV ( Caron et al. 2020 ) , SeLa ( Asano et al. 2020 ) , MSN ( Assran et al. 2022 ) ), and alternative objectives such as redundancy reduction in Barlow Twins ( Zbontar et al. 2021 ) . Another paradigm, inspired by Natural Language Processing (NLP) ( Radford & Narasimhan 2018 ; Devlin et al. 2019 ) , is generative SSL. Examples include iGPT ( Chen et al. 2020a ) , which reconstructs masked pixels, and vision-specific approaches like BEiT ( Bao et al. 2022 ) and MAE ( He et al. 2022 ) , which reconstruct masked patches. More recently, I-JePa ( Assran et al. 2023 ) advanced this concept by predicting embeddings of masked patches using contextual and target encoders. Hybrid approaches, such as iBoT ( Zhou et al. 2022 ) , DINOv2 ( Oquab et al. 2024 ) , combine joint embedding learning with masked image modeling techniques like those