Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models
Zhixiang Wei, Xiaoxiao Ma, Ruishen Yan, Tao Tu, Huaian Chen, Jinjin Zheng, Yi Jin, Enhong Chen · 2025
Read Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models by Zhixiang Wei, Xiaoxiao Ma, Ruishen Yan, Tao Tu, Huaian Chen, Jinjin Zheng, Yi Jin, Enhong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01667.txt.
Introduction
Zhixiang Wei and seven co-authors present Rein++, a framework that adapts large vision foundation models to semantic segmentation, the task of labeling every pixel in an image. Foundation models train on huge data, but segmentation data sets are small and real scenes shift away from training conditions.
One part of the system tunes less than one percent of the model's parameters through small trainable tokens, giving strong generalization from little data. A second part adapts the tuned model to new unlabeled scenes, using a segment anything model to sharpen boundaries. Tests across settings beat prior methods with lean training. The pipeline moves from day scenes to night scenes without new labels.
Catalog record
| CREATOR | Zhixiang Wei, Xiaoxiao Ma, Ruishen Yan, Tao Tu, Huaian Chen, Jinjin Zheng, Yi Jin, Enhong Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01667
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models Zhixiang Wei, Xiaoxiao Ma, Ruishen Yan, Tao Tu, Huaian Chen, Jinjin Zheng, Yi Jin, Enhong Chen Abstract Vision Foundation Models(VFMs) have achieved remarkable success in various computer vision tasks. However, their application to semantic segmentation is hindered by two significant challenges: (1) the disparity in data scale, as segmentation datasets are typically much smaller than those used for VFM pre-training, and (2) domain distribution shifts, where real-world segmentation scenarios are diverse and often underrepresented during pre-training. To overcome these limitations, we present Rein++ , an efficient VFM-based segmentation framework that demonstrates superior generalization from limited data and enables effective adaptation to diverse unlabeled scenarios. Specifically, Rein++ comprises a domain generalization solution Rein-G and a domain adaptation solution Rein-A. Rein-G introduces a set of trainable, instance-aware tokens that effectively refine the VFM’s features for the segmentation task. This parameter-efficient approach fine-tunes less than 1% of the backbone’s parameters, enabling robust generalization. Building on the Rein-G, Rein-A performs unsupervised domain adaptation at both the instance and logit levels to mitigate domain shifts. In addition, it incorporates a semantic transfer module that leverages the class-agnostic capabilities of the segment anything model to enhance boundary details in the target domain. The integrated Rein++ pipeline first learns a generalizable model on a source domain (e.g., daytime scenes) and subsequently adapts it to diverse target domains (e.g., nighttime scenes) without any target labels. Comprehensive experiments demonstrate that Rein++ significantly outperforms state-of-the-art methods with efficient training, underscoring its roles an efficient, generalizable, and adaptive segmentation solution for VFMs, even for large models with billions of parameters. The code is available at https://github.com/wloves/Rein . Fig. 1: This paper proposes an effective training framework for both domain generalization and adaptation. We demonstrate our approach using an experiment with GTAV as source domain and Cityscapes as target domain. (a) Leveraging powerful VFMs and efficient Rein, our method achieves SoTA domain generalization performance with 71.9% mIoU on Cityscapes validation set. (b) By incorporating target domain images, SAM masks, and upgrading Rein to domain-adaptive Rein++ for VFM adaptation, we surpass our generalization results, achieving 78.2% mIoU for SoTA domain adaptation performance. Fig. 2: Using GTAV \rightarrow Cityscapes as a demonstration case for both generalization and adaptation, we applied Rein++ to multiple mainstream VFM families and models of varying scales. Rein++ consistently enhances generalization performance, surpassing both full finetuning and frozen backbone finetuning approaches. Simultaneously, the adaptation process significantly improves these VFMs’ target domain performance. Notably, full finetuning of InternVL (6B) proves exceptionally challenging for semantic segmentation due to its massive parameter size, hence its performance data is unavailable. By contrast, Rein++ requires training only 0.4% of its parameters, enabling execution with approximately 24GB GPU memory while delivering substantially improved performance over frozen variants. This demonstrates Rein++’s advantage for training large models. I Introduction The emergence of large-scale Vision Foundation Models (VFMs), exemplified by models like CLIP [ 1 ] , DINOv2 [ 2 ] , and InternVL-c [ 3 ] , has marked a paradigm shift in computer vision, demonstrating robust generalization across numerous tasks. However, their application to semantic segmentation presents two significant challenges. First, the disparity in dataset scale: Semantic segmentation datasets, such as Cityscapes [ 4 ] and GTAV [ 5 ] , are orders of magnitude smaller than the web-scale datasets ( e.g. , LVD-142M [ 2 ] ) used for VFM pre-training. Naively fine-tuning VFMs, which have a vast number of trainable parameters (ranging from approximately 300 million to 6 billion), on these smaller datasets can result in limited generalizability [ 6 ] . Second, the domain distribution shift: Real-world segmentation scenarios are highly diverse, encompassing challenging conditions such as rain, fog, nighttime, and snow, which are underrepresented in the pre-training datasets of VFMs [ 7 ] . Furthermore, annotating data for these diverse conditions is significantly more challenging than for a uniform and well-defined setting. Consequently, adapting VFMs to multiple domains without labeled data becomes a difficult and significant task. For traditional networks such as VGG [ 8 ] and ResNet [ 9 ] , these challenges have been effectively addressed. The issue of dataset scale disparity is tackled by Domain Generalization Semantic Segmentation (DGSS) methods [ 10 , 11 , 12 , 13 , 14 , 15 , 16 ] , which train models on source domain datasets and enhance their generalization ability across unseen domains. Meanwhile, the domain distribution shift is addressed by unsupervised Domain Adaptive Semantic Segmentation (DASS) methods [ 17 , 18 , 19 , 20 , 21 ] , which leverage images from the target domain to improve performance under domain shift. However, these strategies have predominantly utilized CNN backbones [ 9 , 22 ] or compact ViT [ 23 ] , and have not fully exploited the potential of billion-parameter VFMs. To address this gap, we propose an efficient VFM-based segmentation framework that demonstrates superior generalization capabilities when fine-tuned on limited datasets, while also enabling effective adaptation to diverse, unlabeled scenarios. For Efficiency: To utilize VFMs efficiently in limited datasets, we opt to fine-tune VFMs with a reduced set of trainable parameters, i.e. , parameter efficient fine-tuning (PEFT) methods. A straightforward approach involves freezing the VFMs and fine-tuning only the decoder head. Our experiments, as shown in Table I , reveal surprising results: even with a frozen backbone, VFMs outperform most existing DG methods in terms of generalizability. However, since VFMs are typically pre-trained on tasks unrelated to segmentation, they lack task-specific knowledge for segmentation contexts such as urban scenes. Despite the efficiency, backbone freezing thus fails to fully exploit VFMs’ segmentation potential. For Generalization: To enhance generalization while maintaining efficiency, we propose Rein-G , a robust PEFT method specifically designed for segmentation. Rein-G utilizes a minimal set of trainable parameters to leverage VFMs for superior generalization. At its core, Rein-G incorporates a set of learnable tokens, each directly associated with different instances. These tokens interact with VFM features through a dot-product operation to generate an attention-like similarity map. This map enables Rein-G to precisely refine and propagate the feature maps from each layer to the next within the backbone, significantly boosting the performance of VFMs in segmentation scenarios. Through deliberate design and token-level refinement, Rein-G achieves superior generalization compared to existing PEFT methods [ 24 , 25 , 26 ] . For Adaptation: While Rein-G demonstrates strong generalizability, it lacks the mechanism to adapt to unlabeled domains. To overcome this limitation, we propose Rein-A , which extends Rein-G by leveraging its core strength: the effective encapsulation of source domain knowledge within a compact set of parameters. This enables efficient adaptation by fine-tuning only these essential components rather than the entire parameter set. Rein-A introduces three key innovations: 1) Mask classification adaptation segmentation: Diverging from conventional per-pixel classification [ 18 , 19 , 17 , 27 , 28 , 29 ] , Rein-A aligns with modern VFM segmentation architectures [ 30 , 31 , 32 , 33 ] by formulating adaptation through mask-based objectives. 2) Dual-level supervision: Rein-A employs complementary loss functions at both the semantic logit level and instance mask level, providing more precise gradient signals than single-level approaches and resulting in stabler convergence. 3) Rein-A integrates