FuTCR: Future-Targeted Contrast and Repulsion for Continual Panoptic Segmentation

Nicholas Ikechukwu, Keanu Nichols, Deepti Ghadiyaram, Bryan A. Plummer · 2026

Read FuTCR: Future-Targeted Contrast and Repulsion for Continual Panoptic Segmentation by Nicholas Ikechukwu, Keanu Nichols, Deepti Ghadiyaram, Bryan A. Plummer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12451.txt.

Introduction

Ikechukwu and three colleagues fix a quiet flaw in continual panoptic segmentation, where new object classes arrive over time. Training images mix labelled and unlabelled objects, and older methods lump every unlabelled pixel into one background class. That teaches the model that distinct future classes are all the same thing, making them hard to learn later. Their FuTCR framework reshapes representations before new classes appear, first finding confident future-like regions from grouped masks, then pushing them apart so new classes start distinct.

Catalog record

CREATORNicholas Ikechukwu, Keanu Nichols, Deepti Ghadiyaram, Bryan A. Plummer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50943
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12451

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FuTCR: Future-Targeted Contrast and Repulsion for Continual Panoptic Segmentation Nicholas Ikechukwu, Keanu Nichols, Deepti Ghadiyaram, Bryan A. Plummer Abstract Continual Panoptic Segmentation (CPS) requires methods that can quickly adapt to new categories over time. The nature of this dense prediction task means that training images may contain a mix of labeled and unlabeled objects. As nothing is known about these unlabeled objects a priori , existing methods often simply group any unlabeled pixel into a single "background" class during training. In effect, during training, they repeatedly tell the model that all the different background categories are the same (even when they aren’t). This makes learning to identify different background categories as they are added challenging since these new categories may require using information the model was previously told was unimportant and ignored. Thus, we propose a Future-Targeted Contrastive and Repulsive (FuTCR) framework that addresses this limitation by restructuring representations before new classes are introduced. FuTCR first discovers confident future-like regions by grouping model-predicted masks whose pixels are consistently classified as background but exhibit non-background logits. Next, FuTCR applies pixel-to-region contrast to build coherent prototypes from these unlabeled regions, while simultaneously repelling background features away from known-class prototypes to explicitly reserve representational space for future categories. Experiments across six CPS settings and a range of dataset sizes show FuTCR improves relative new-class panoptic quality over the state-of-the-art by up to 28% , while preserving or improving base-class performance with gains up to 4% 1 1 1 The models and code will be made publicly available: https://github.com/futcr/FuTCR . 1 Introduction Continual learning aims to expand a model’s recognition capability by introducing new categories over time without retraining from scratch [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] . Most prior work studies classification, where the model updates only a global label predictor [ 2 , 9 , 11 , 12 , 13 , 14 , 15 ] , where samples only contain examples of labeled classes. In contrast, dense prediction tasks like image segmentation may contain a mix of labeled and unlabeled categories [ 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ] . As shown in Figure 1 (a), many of these methods map all unlabeled pixels into a single "background" category. Thus, these methods may only learn how to identify it as any unlabeled category, rather than learning how to distinguish between objects. For example, if no vehicles are labeled, then a model could just look for a wheel to mark that object as background. Thus, there would be no need to learn to separate cars and trucks even though they may appear as new categories to learn in the future. Figure 1: From background noise to structured supervision . (a) Prior continual panoptic methods treat background regions as non-informative [ 21 , 23 , 29 , 30 ] , allowing future-class evidence to be absorbed into existing decision regions in feature space over time. (b) Our framework instead converts background activations into future-aware structural cues that organize scene composition before new labels arrive, reducing semantic drift and improving Panoptic Quality (PQ). We address this limitation by proposing a Fu ture- T argeted C ontrastive and R epulsive ( FuTCR ) learning framework that explicitly restructures the representation space before new classes are introduced. As illustrated in Figure 1 (b), our core idea is to treat unlabeled content that is likely to belong to future classes not as undifferentiated background, but as structured regions that should remain separable from known-class representations. To do so, FuTCR uses the model’s own panoptic predictions at the base step to discover future-like regions, i.e . , connected mask regions that are consistently predicted as background yet exhibit non-background evidence in the logits. Within each discovered region, FuTCR applies a pixel-to-region contrastive objective that pulls pixels toward a shared region prototype and pushes them away from prototypes of other regions. This objective enforces coherence inside each unlabeled region and diversity across regions, encouraging the model to form distinct, reusable prototypes for future categories instead of collapsing them into background. To further prevent unlabeled regions from being absorbed into known-class decision regions, FuTCR introduces a future-targeted repulsion objective . Concretely, we maintain a set of prototypes for known classes and explicitly repel features from discovered future-like regions away from these prototypes in both feature and logit space. This repulsion discourages future evidence from occupying the same representational subspace as old classes and creates a margin around known-class prototypes that can later be allocated to new categories. Together, the contrastive and repulsive terms cause the current-step model to pre-organize its feature space so that future-class pixels form coherent clusters that are distinct from old classes, which eases the integration of new categories during incremental updates. Our future-aware error and representation analysisconfirms that FuTCR significantly reduces future-old class confusion compared to strong baselines. While prior work has explored using unlabeled data via semi-supervised or self-supervised objectives [ 31 , 32 , 33 , 34 ] , these methods primarily improve performance on currently labeled classes and do not explicitly shape the feature space for future class expansion in continual segmentation. Recent work on future-aware representation learning for segmentation [ 35 ] moves in this direction but relies on externally defined regions or non-adaptive clustering, which decouples region construction from the model’s evolving predictions. In contrast, FuTCR couples region discovery, representation learning, and classifier preparation in a single model-driven procedure: predicted masks define the regions, contrastive learning structures the corresponding feature clusters, and repulsion from known-class prototypes reserves capacity for future categories. This tight coupling yields a scalable, adaptive mechanism for organizing representations ahead of future class introductions, resulting in a relative gain of up to 28% on new classes coupled with gains up to 4% on old categories. In summary, our main contributions are: • We identify a fundamental extensibility limitation in continual panoptic segmentation: representations and classifier parameters become progressively biased toward previously learned categories, suppressing future-class integration under background overlap. • We propose FuTCR , a future-targeted region-level unsupervised learning framework that leverages model-predicted masks to construct supervision from unlabeled regions, combining contrastive structuring and known-class repulsion to reserve explicit capacity for future categories. • Through a dedicated future-aware error and representation analysis, we show that FuTCR reshapes the feature space to reduce future-known class confusion, leading to consistent gains in novel-class panoptic quality across two extreme settings, including fully shared and fully disjoint image overlaps, while preserving performance on previously learned classes. 2 Related Work Recent continual panoptic segmentation methods typically adapt dense prediction architectures to class-incremental streams by combining mask-based distillation [ 21 , 23 , 36 ] , pseudo-labeling [ 29 , 36 ] , replay [ 30 , 29 , 37 ] , or prompt tuning [ 28 , 38 ] to alleviate catastrophic forgetting [ 39 , 40 , 41 ] and maintain panoptic quality [ 29 , 21 , 30 , 23 , 28 , 42 ] . These approaches successfully preserve performance on previously learned classes, but they still

Read the complete text