Prototypical Progressive Alignment and Reweighting for Generalizable Semantic Segmentation

Yuhang Zhang, Zhengyu Zhang, Muxin Liao, Shishun Tian, Wenbin Zou, Lu Zhang, Chen Xu · 2025

Read Prototypical Progressive Alignment and Reweighting for Generalizable Semantic Segmentation by Yuhang Zhang, Zhengyu Zhang, Muxin Liao, Shishun Tian, Wenbin Zou, Lu Zhang, Chen Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-11955.txt.

Introduction

Yuhang Zhang and co-authors improve image labeling that must work on unseen scenes, in a paper from two thousand twenty-five. Class prototypes, the average feature of each class, offer stable cues across settings. Past methods align them crudely, build them from noisy batches, and weigh all training images alike.

Their frame pairs two text-based prototypes drawn from a vision-language model. Alignment then proceeds from easy to hard cases, shrinking the scene gap step by step. A reweighting step downplays unreliable training images to block negative transfer. Theory support and wide benchmark wins back the design, the authors report.

Catalog record

CREATORYuhang Zhang, Zhengyu Zhang, Muxin Liao, Shishun Tian, Wenbin Zou, Lu Zhang, Chen Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-11955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prototypical Progressive Alignment and Reweighting for Generalizable Semantic Segmentation Yuhang Zhang, Zhengyu Zhang, Muxin Liao, Shishun Tian, Wenbin Zou, Lu Zhang, Chen Xu Abstract Generalizable semantic segmentation, aims to excel on unseen target domains, as a critical focus due to the widespread practical applications requiring high generalizability. Class-wise prototypes, which depict class-wise centroids, as a type of domain-invariant information are key to improving the model generalizability due to its stability and representativeness. However, this manner faces some challenges. First, the existing methods adopt a coarse prototypical alignment form, potentially compromising performance. Second, the naive prototype generally serves as the class centroid generated by an average operation from source data batches, risks source domain overfitting, and may be detrimentally impacted by unrelated source data. Third, from a broader perspective, rather than just from a prototypical alignment perspective, the existing methods treat all samples equally, which is against the conclusion that different source features have different adaptation difficulties. To tackle these issues, we propose a novel method for generalizable semantic segmentation called Prototypical Progressive Alignment and Reweighting (PPAR) depending on the strong generalized representation of the Contrastive Language-Image Pretraining (CLIP) model. In particular, we first define the Original Text Prototype (OTP) and Visual Text Prototype (VTP) generated by the CLIP model, laying the foundation for the subsequent effective alignment strategy. Then, we propose a prototypical progressive alignment strategy by an easy-to-difficult alignment form to reduce domain-variant information progressively instead of directly. Finally, we propose a prototypical reweighting learning strategy that estimates the importance of the source data and corrects its learning weight to alleviate the influence of unrelated source features, i.e. alleviate negative transfer. Moreover, we also offer a theoretical insight into our method and it shows that our method compiles well on the domain generalization theory. Extensive experiments on several popular datasets demonstrate that our PPAR method achieves superior performance, proving the effectiveness of our method. I Introduction Intelligent Transportation Systems (ITSs) is a comprehensive field focusing on improving transportation effectiveness, widely-used in automatic driving, moving robotics, and so on. These ITSs heavily rely on perception tasks, which are vital for interpreting the surrounding environment and making informed decisions. Among perception tasks, semantic segmentation [ 1 ] plays a crucial role in accurately identifying objects such as vehicles, pedestrians, and traffic signs, ensuring the system can operate safely and effectively, which aims to give the correct category label to each pixel of images [ 2 , 3 ] . Recently, several types of popular paradigms have been made by researchers. As a typical manner, supervised semantic segmentation [ 4 ] has achieved superior performance without considering domain divergence between the training and validation sets, which is kindly applied in unchanged environments while suffering a performance drop in changed environments due to the domain shift problem [ 5 ] (i.e., there is a domain divergence between the training and validation sets). Limited by the massive cost of data labeling [ 6 ] and the inability to tackle the domain shift problem [ 7 ] , unsupervised domain adaptation (UDA) was developed as a new task [ 8 , 9 ] , which aims to achieve remarkable performance in the target domain given the labeled source data and unlabeled target data. The domain gap, which harms the performance in the target domain, is reduced by some efficient UDA strategies such as distribution alignment [ 9 ] and image style transfer [ 10 ] . Nevertheless, the available target data is a primary obstruction in practical applications since many applications have privacy protection requirements. Moreover, performing well in one fixed target environment can not satisfy practical requirements owing to the diversity in the real world. Figure 1: Approach comparison. The red ellipsoid denotes the source domain, the green one represents the overarching unknown target domain, while the yellow and blue ellipsoids indicate specific target domains. In prototypical reweighting, the red percent in features represents the learning weight. The source outlier has a relatively low learning weight. More recently, domain generalization (DG) has attracted a lot of attention to tackle these above drawbacks in UDA, with the goal of generalizing well on all probable unseen domains [ 11 ] . Different from the UDA, the target domain data do not participate in the training period of DG and is only used for evaluation [ 12 ] . Designing a DG model with high generalizability generally obeys a core rule that extracts domain-invariant features, which can be roughly devised from two aspects, i.e., style extension and style erasion. Style extension [ 13 , 14 ] generates images with different styles as the extended training set to enhance the data diversity. Style erasion uses some representation constraints like normalization [ 15 ] , whitening [ 16 ] , and prototypical alignment [ 17 ] . Among them, prototypical alignment is an effective method for domain adaptation and domain generalization, which pulls close the class-wise features to the related class prototype (i.e., class centroid) [ 18 ] . In the prototypical alignment paradigm, the class-wise prototype is expected as the class-wise domain-invariant feature since that can be seen as the combined representation of multi-domains. To learn domain-invariant features from the class prototype, three key observations emerge as follows. 1. The naive prototype can be regarded as the source-bias prototype that provides a weak generalizable representation for prototypical alignment, as typical approaches [ 17 , 19 , 20 ] generate the prototype by computing the class centroid from a batch of source data rather than using overall domain data. The overall domain denotes the potential domain consisting of the source and many unseen sub-target domains, which is represented by the green ellipsoid in Fig. 1 . It can be seen that the naive prototype (i.e., red circle) is generated by several source samples, which can only reflect the source centroid, rather than the overall centroid (i.e., green circle) that contains more generalizable information. Aligning features with such naive prototypes may drive the model towards source-biased prototypes, increasing the risk of overfitting to the source domain and susceptibility to source outliers. 2. The existing prototypical alignment methods can be broadly categorized into single-layer prototypical alignment (SPA) and multi-layer prototypical alignment (MPA), as illustrated in Fig. 1 . SPA performs alignment only at a specific layer, typically the final one [ 17 ] , focusing on a single-layer representation. However, this approach results in insufficient alignment, leading to suboptimal performance, as it fails to consider the multi-scale alignment of features. In contrast, MPA attempts to align features across multiple layers, capturing deeper information but often neglecting the unique attributes of shallow features [ 19 ] , such as their ability to capture basic visual information or style cues [ 21 , 22 ] . Although multi-scale alignment methods have demonstrated better performance [ 23 , 24 ] , MPA’s uniform treatment of shallow and deep features can inadvertently remove useful information from the shallow layers, which play a crucial role in encoding fundamental visual details. An refined approach is required to fully leverage both shallow and deep features without discarding valuable low-level information, thus addressing the limitations of both SPA and MPA. The rectangle with rounded corners and dashed edges of

Read the complete text