Rethinking Query-based Transformer for Continual Image Segmentation

Yuchen Zhu Cheng Shi Dingyou Wang Jiajin Tang Zhengxuan Wei, Yu Wu Guanbin Li Sibei Yang · 2025

Read Rethinking Query-based Transformer for Continual Image Segmentation by Yuchen Zhu Cheng Shi Dingyou Wang Jiajin Tang Zhengxuan Wei, Yu Wu Guanbin Li Sibei Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07831.txt.

Introduction

Teaching a photo labeler class by class makes it forget old classes, unless its mask sense is protected. Yuchen Zhu and co-authors rethink that protection in a 2025 paper. Query-based transformers carry a built-in feel for objects.

Split designs that guard that feel lose flexibility and lean on data order. Rich image features let queries fake masks by simple matching, they find. Their method picks features directly for each query instead. New classes stay learnable while old masks hold. A replay trick with visual queries guards old classes further. Tests across tasks and orders beat prior methods.

Catalog record

CREATORYuchen Zhu Cheng Shi Dingyou Wang Jiajin Tang Zhengxuan Wei, Yu Wu Guanbin Li Sibei Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07831

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Query-based Transformer for Continual Image Segmentation Yuchen Zhu Cheng Shi Dingyou Wang Jiajin Tang Zhengxuan Wei, Yu Wu Guanbin Li Sibei Yang Abstract Class-incremental/Continual image segmentation (CIS) aims to train an image segmenter in stages, where the set of available categories differs at each stage. To leverage the built-in objectness of query-based transformers, which mitigates catastrophic forgetting of mask proposals, current methods often decouple mask generation from the continual learning process. This study, however, identifies two key issues with decoupled frameworks: loss of plasticity and heavy reliance on input data order. To address these, we conduct an in-depth investigation of the built-in objectness and find that highly aggregated image features provide a shortcut for queries to generate masks through simple feature alignment. Based on this, we propose SimCIS, a simple yet powerful baseline for CIS. Its core idea is to directly select image features for query assignment, ensuring “perfect alignment” to preserve objectness, while simultaneously allowing queries to select new classes to promote plasticity. To further combat catastrophic forgetting of categories, we introduce cross-stage consistency in selection and an innovative “visual query”-based replay mechanism. Experiments demonstrate that SimCIS consistently outperforms state-of-the-art methods across various segmentation tasks, settings, splits, and input data orders. All models and codes will be made publicly available at https://github.com/SooLab/SimCIS . 1 Introduction Continual learning empowers models to progressively acquire, learn, and assimilate new knowledge from an ever-evolving environment. It serves as a fundamental task in image classification Chaudhry et al. 2018a ; Douillard et al. 2020 ; Rebuffi et al. 2017 ; Castro et al. 2018 ; Mallya and Lazebnik 2018 ; Zhang et al. 2023a ; Yan et al. 2021 ; Robins 1995 ; Shin et al. 2017 ; Shi et al. 2024a ; Dai and Yang 2024 ; Shi and Yang 2024 ; Tang et al. 2023b ; Lin et al. 2021 ; Yang et al. 2021 ; Ge et al. 2018 ; He et al. 2019 where models are required to recognize new classes ( plasticity ) and preserve old class knowledge (avoid catastrophic forgetting ). Extending beyond classification, continual image segmentation adapts this to the image segmentation, unlocking a myriad of practical applications Rusu et al. 2016b ; Ross et al. 2008 . However, it also confronts more challenges: 1) Additional catastrophic forgetting of mask prediction, beyond that of class prediction; 2) Background semantic shift occurs when the current foreground becomes background in subsequent stages, driven by the need for image segmentation to predict the background class and the constraint of only having class annotations from current stage. Figure 1 : Boxplots of PQ metric for our SimCIS and previous SOTA Kim et al. 2024 on ADE20K. We train each model on randomly shuffled continual data input orders and report average PQ for base and novel classes. We observe that recent query-based transformers suffer from a loss of plasticity (low average PQ) and heavy reliance on the input data order (high variance). Recently, query-based transformers Cheng et al. 2022 ; Chen et al. 2024a ; Tang et al. 2023a ; Shi et al. 2024b ; Huang et al. 2023 ; Zheng et al. 2023 ; Huang et al. 2025 ; Shi and Yang 2023b ; Shi and Yang 2023a are introduced into continual image segmentation, as their built-in objectness has been shown to mitigate catastrophic forgetting in mask generation. Leveraging this built-in objectness, many studies Kim et al. 2024 ; Gong et al. 2024 ; Cha et al. 2021a ; Zhang et al. 2022a decouple mask segmentation from the continual learning process by freezing the parameters associated with mask proposal generation. However, we observe two notable yet suboptimal behaviors in the aforementioned methods. • The advantage of objectness diminishes and even has a detrimental effect on plasticity as the task sequence shortens. In the shortest two-task setting, they typically achieve performance comparable to or even slightly lower than the baseline. • The built-in objectness is fragile and lacks robustness, showing heavy dependence on the split and order of input data. As shown in Fig 1 , in ten random trials, the worst trial shows a significant performance drop on new classes compared to the default setting. Therefore, in this work, we aim to understand the built-in objectness and achieve consistent improvements (especially on plasticity) across different task lengths and varying data input orders. This is crucial, as it is impractical to assume fixed task lengths and data sequences in real-world scenarios. Figure 2 : Clustering results from feature map. Pixel feature provides sufficient semantic priors (Person) even after finetuning. The conclusion from a series of investigations is: • ❶ The built-in objectness emerges from the alignment between the query and the semantic priors within the image feature, mediated by the decoder. As shown in Fig 2 , the clustering results indicate that image features contain sufficient semantic priors where pixels belonging to the same semantic are grouped together) even after finetune. Meanwhile, the query continuously aligns with specific regions of the feature map at each layer of the decoder as shown in Fig 3 (right). In summary, the highly aggregated image feature provides a shortcut for queries to generate masks by simply aligning themselves to semantic priors in the image feature through the decoder. • ❷ The built-in objectness diminishes over training stages due to the query’s failure to align with the semantic priors of the feature map. As shown in Fig 3 (left), since semantic priors vary at different stages due to background semantic shift, causing the updated learnable query to gradually misalign with the pixel feature from old classes in previous stages, even after the decoder’s post-alignment (observed in ❶ ). Inspired by ❶ and ❷ , to ensure objectness is preserved throughout the continual learning stages, we propose a lazy Query Pre-Alignment (QPA) method, where query features are selected from specific locations in the image feature map, rather than being learned from scratch, to “ perfectly ” pre-align query feature with semantic priors. Specifically, based on the current stage’s semantic classes, we select the most semantically significant locations in the image feature, preserving objectness at each stage. However, objectness is still lost across stages due to varying semantic classes in different stages. To overcome cross-stage selection issues, a naive solution involves distillation on the feature map or query features between stages. However, in turn, while it preserves old priors from previous stages, it re-introduces incorrect priors for current stages (where old priors label current semantics as background), leading to a loss of plasticity. Fortunately, thanks to our query pre-alignment method, we can easily maintain old classes by keeping queries corresponding to old class positions, while enabling the selection of remaining queries for new classes in the current stage. Thus, we propose a Consistent Selection Loss (CSL) to ensure that, for the same image, the most semantically significant locations selected in the previous stage are revisited in the current stage. With QPA and CSL, objectness in the query-based transformer is fully utilized to generate mask proposals. However, for class prediction, catastrophic forgetting may still occur. Previous methods typically rely on image replay to mitigate catastrophic forgetting. In contrast, thanks to our query pre-alignment, our query inherently contains category semantics.

Read the complete text