Probabilistic Prototype Calibration of Vision-Language Models for Generalized Few-shot Semantic Segmentation
Jie Liu, Jiayi Shen, Pan Zhou, Jan-Jakob Sonke, Efstratios Gavves · 2025
Read Probabilistic Prototype Calibration of Vision-Language Models for Generalized Few-shot Semantic Segmentation by Jie Liu, Jiayi Shen, Pan Zhou, Jan-Jakob Sonke, Efstratios Gavves (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22979.txt.
Introduction
Liu, Shen, Zhou, Sonke, and Gavves tune the class templates of vision-language models with odds in mind, for pixel labeling from a few shots, in work from twenty twenty-five. The task adds new classes from a few example shots, while base classes stay sharp. Fixed, one-value templates bend poorly to new shots that swing a lot, when labels are scarce. Their FewCLIP frame tunes CLIP templates with a sense of doubt built in.
Frozen text templates are honed by visual templates that learn. A rule keeps the spread of tuned templates in shape, so they do not just memorize the few shots. That shaped doubt helps the model carry to fresh swings in new classes. On the PASCAL and COCO sets, FewCLIP beats the best past scores by a clear gap. The gains hold when classes are added one by one, and on the plain splits.
Catalog record
| CREATOR | Jie Liu, Jiayi Shen, Pan Zhou, Jan-Jakob Sonke, Efstratios Gavves |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22979
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Probabilistic Prototype Calibration of Vision-Language Models for Generalized Few-shot Semantic Segmentation Jie Liu, Jiayi Shen, Pan Zhou, Jan-Jakob Sonke, Efstratios Gavves Abstract Generalized Few-Shot Semantic Segmentation (GFSS) aims to extend a segmentation model to novel classes with only a few annotated examples while maintaining performance on base classes. Recently, pretrained vision-language models (VLMs) such as CLIP have been leveraged in GFSS to improve generalization on novel classes through multi-modal prototypes learning. However, existing prototype-based methods are inherently deterministic, limiting the adaptability of learned prototypes to diverse samples, particularly for novel classes with scarce annotations. To address this, we propose FewCLIP, a probabilistic prototype calibration framework over multi-modal prototypes from the pretrained CLIP, thus providing more adaptive prototype learning for GFSS. Specifically, FewCLIP first introduces a prototype calibration mechanism, which refines frozen textual prototypes with learnable visual calibration prototypes, leading to a more discriminative and adaptive representation. Furthermore, unlike deterministic prototype learning techniques, FewCLIP introduces distribution regularization over these calibration prototypes. This probabilistic formulation ensures structured and uncertainty-aware prototype learning, effectively mitigating overfitting to limited novel class data while enhancing generalization. Extensive experimental results on PASCAL-5 i and COCO-20 i datasets demonstrate that our proposed FewCLIP significantly outperforms state-of-the-art approaches across both GFSS and class-incremental setting. The code is available at https://github.com/jliu4ai/FewCLIP . 1 Introduction Semantic segmentation [ 4 , 47 , 5 ] aims to assign semantic labels to each pixel of an image and has achieved remarkable success in the deep learning era. However, traditional semantic segmentation heavily relies on vast amounts of data with pixel-wise annotations and often struggles with classes absent from the training set, a.k.a. novel classes. To tackle these challenges, few-shot semantic segmentation (FSS) [ 45 , 40 , 37 , 3 , 42 , 7 , 26 , 23 , 28 , 44 , 31 ] has emerged. FSS first trains a base model on many annotated data of base classes to learn general knowledge, and then fine-tunes the model on a few annotated samples of novel classes for adaptation. However, FSS primarily focuses on the binary segmentation of novel classes at test time, but lacks identification ability on base classes and multi-novel classes [ 25 , 12 ] . Figure 1 : Concept diagram comparing deterministic and probabilistic prototype learning with pre-trained CLIP in GFSS task. (a) Existing methods integrate a learnable or precomputed visual prototype with a frozen textual prototype for each class to perform pixel-wise deterministic classification. (b) Our proposed FewCLIP models the class-wise visual calibration prototype as a distribution, capturing intra-class variations and uncertainty. By representing prototypes with a probabilistic distribution, our method mitigates prototype overfitting to scarce data, thereby improving generalization to novel classes. As a more practical and ambitious extension, generalized few-shot semantic segmentation (GFSS) [ 38 , 25 , 12 , 29 , 9 , 10 , 34 , 39 ] aims to enable the segmentation of both base and novel classes within a given image. Typically, GFSS consists of two phases: (1) the base class training phase , where a base segmentation model is trained on base classes with abundant annotated samples, and (2) the novel class registration phase , where novel classes are integrated into the model using only a few annotated samples per class. Due to the significant data availability gap between these two phases, the primary challenge in GFSS lies in enhancing model generalization to novel classes while preserving performance on base classes . Generally, existing methods [ 12 , 25 , 43 ] tackle this challenge by decoupling the prototype learning of base and novel classes, ensuring that only novel class prototypes are optimized during the novel class registration phase. More recently, O3S [ 43 ] introduced multi-modal prototypes learning with pre-trained CLIP [ 32 ] to further improve model generalization on novel classes. However, as shown in Fig. 1 (a), existing prototype-based methods remain deterministic, failing to account for the inherent uncertainty in learned visual prototypes. For instance, images from the same class can exhibit significant variations in appearance, pose, and orientation, making it challenging for a deterministic prototype to adequately capture all possible variations. As a result, overlooking uncertainties in prototype learning can lead to overfitting on the limited training samples and causing the model to forget generalizable knowledge [ 18 , 13 ] . This issue is particularly severe during the novel class registration phase, where visual prototypes must be learned from only a few annotated samples, further hindering model generalization. Contributions. To address these limitations, we propose FewCLIP, a novel probabilistic prototype calibration framework that integrates uncertainty-aware prototype learning with pretrained vision-language models (VLMs). To enhance generalization to novel classes, we first introduce a prototype calibration mechanism, where FewCLIP calibrates frozen textual prototypes from pre-trained CLIP with learnable visual calibration prototypes in a class-specific manner. This mechanism preserves the generalizable knowledge from CLIP while incorporating class-specific information by learning calibration prototypes from supervised data. By optimizing only the visual prototypes for novel classes during the novel class registration phase, FewCLIP ensures effective adaptation to novel classes while preserving base class knowledge. Furthermore, to mitigate the overfitting of calibration prototypes on limited observed samples—especially in the novel class registration phase, we introduce probabilistic prototype calibration. As illustrated in Fig. 1 (b), instead of representing calibration prototypes as deterministic vectors, we model them as continuous probability distributions, allowing them to capture uncertainty and intra-class variation in the learned representations. This probabilistic formulation provides a distribution regularization over visual calibration prototypes, mitigating overfitting to limited novel class samples, thereby improving model generalization over these classes. In summary, our contributions are threefold: \bullet We introduce a prototype calibration mechanism for GFSS, adapting frozen textual prototypes with visual calibration prototypes to enhance model generalization to novel classes while maintaining performance on base classes. \bullet We propose FewCLIP, a probabilistic prototype calibration framework modeling calibration prototypes as probability distributions to mitigate prototype overfitting and enable uncertainty-aware prototype learning. \bullet Extensive experiments on PASCAL-5 i and COCO-20 i benchmarks demonstrate the superiority of our method in both GFSS and class-incremental FSS settings, achieving around 10% mIoU improvement on novel classes. 2 Related Work Generalized Few-shot Semantic Segmentation (GFSS). Unlike conventional few-shot semantic segmentation, GFSS [ 38 ] aims to recognize both base and novel classes in an image, leveraging sufficient data from base classes and a few data from novel classes. One line of research in GFSS seeks to mimic the novel class registration during base class training, employing techniques such as contextual aggregation [ 38 ] , orthogonality [ 25 ] , and kernel-based learning [ 12 ] . These methods enable a seamless extension of the label space when incorporating novel classes. Another research direction explores various fine-tuning [ 29 , 9 , 10 ] strategy during novel class registration to effectively adapt models to novel classes. More recently, BCM [ 34 ] offer a strong baseline for GFSS by training novel-class models by supervised learning. O3S [ 43 ] and BCM [ 39 ] investigate how multi-modal knowledge from vision-language models can enhance model generalization to novel classes. Vision-Language Models (VLMs). Pre-trained Vision-Language Models (VLMs) [ 14 , 21 , 32 ] connect the image representation with text embedding and have made