Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Models PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE
Xinhui Li, Xinyu He, Qiming Hu, Xiaojie Guo · 2025
Read Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Models PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE by Xinhui Li, Xinyu He, Qiming Hu, Xiaojie Guo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01582.txt.
Introduction
Xinhui Li, Xinyu He, Qiming Hu, and Xiaojie Guo recast domain shift learning around giant vision models in a 2025 paper. Classic domain generalization trains a model to face unseen settings blind. Giant models pretrained on vast mixed data blur that premise, since the target world may already sit inside their training stock.
Their frame, Set Pivot Learning, treats the pretrained stock as a pivot to refine from, not a wall to cross. It favors live, task driven tuning that keeps broad strength while sharpening for the job at hand. Their tuning method pairs a class aware prompt builder with a feature focuser led by those prompts. Tests on standard benches beat top past methods. The authors pitch the frame as the right base for the foundation model era.
Catalog record
| CREATOR | Xinhui Li, Xinyu He, Qiming Hu, Xiaojie Guo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01582
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Models PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE Xinhui Li, Xinyu He, Qiming Hu, Xiaojie Guo Abstract In this paper, we introduce, for the first time, the concept of Set Pivot Learning, a paradigm shift that redefines domain generalization (DG) based on Vision Foundation Models (VFMs). Traditional DG assumes that the target domain is inaccessible during training, but the emergence of VFMs, trained on vast and diverse data, renders this assumption unclear and obsolete. Traditional DG assumes that the target domain is inaccessible during training, but the emergence of VFMs, which are trained on vast and diverse datasets, renders this assumption unclear and obsolete. To address this challenge, we propose Set Pivot Learning (SPL), a new definition of domain migration task based on VFMs, which is more suitable for current research and application requirements. Unlike conventional DG methods, SPL prioritizes adaptive refinement over rigid domain transfer, ensuring continuous alignment with evolving real-world conditions. Specifically, SPL features two key attributes: (i) Dynamic adaptation, transitioning from static domain alignment to flexible, task-driven feature optimization, enabling models to evolve with downstream scenarios; (ii) VFM-centric tuning, leveraging pretrained knowledge as a pivot to hone task-specific representations while preserving cross-domain robustness. Building on SPL, we propose a Dynamic Prompt Fine-Tuning method, which combines a Dynamic Class-aware Prompter with a Prompt-guided Feature Focuser, to elevate VFM performance in targeted scenarios. Extensive experiments on benchmark datasets show the effectiveness of our method, highlighting its superiority over state-of-the-art methods, particularly in generalized segmentation. I Introduction Recent advances in vision foundation models (VFMs) have exhibited exceptional performance across a wide spectrum of computer vision tasks, demonstrating their superior robustness and generalization capabilities. Benefiting from large-scale and multimodal pretraining, VFMs have established new benchmarks across various tasks, such as image classification [ 1 ] , object detection [ 2 , 3 ] , and semantic segmentation [ 4 , 5 ] . Building on these successes, recent research has begun to explore VFMs in more complex and unpredictable scenarios. Among these, generalized segmentation stands out, as it challenges models with real-world variability, including unknown weather conditions, dynamic environments, and a diverse array of street objects. Domain generalization (DG) has long been proposed to address these challenges by defining tasks with strict domain boundaries, training models on source domains and evaluating them on entirely unseen target domains. Traditional DG methods [ 6 , 7 , 8 , 9 ] , often built upon classical network architectures, enhance generalization through data augmentation or intrinsic feature learning to mitigate distribution shifts in unseen data. However, with the emergence of VFMs, the rigid domain partitioning central to DG has grown increasingly ambiguous. As illustrated in Fig. 2 , VFMs undergo pre-training on vast and diverse data, making it likely that target domains are indirectly encountered or even partially supervised. This possibility raises concerns about information leakage and challenges the foundational assumptions of DG frameworks. Simultaneously, the range of observed distributions in VFM-based models introduces unique challenges for generalization tasks, distinct from those faced by traditional DG methods. As shown in Fig. , the primary challenges for downstream tasks have shifted from feature transfer to feature refinement and refocus. While VFMs are pretrained on large-scale, diverse datasets with strong generalization capabilities, they are not optimized for specific tasks. Striking a balance between the generalization strengths of Vision Foundation Models and their task-specific adaptability, such as addressing distributional discrepancies, designing effective model fine-tuning strategies, and efficiently adapting to evolving tasks, remains an open challenge. Based on these insights, we revisit the domain generalization task and raise two key questions: (1) How should the traditional DG problem be redefined when utilizing VFM models, given the potential risk of target domain data leakage? and (2) In the face of feature focus challenge, how can we effectively harness the generalization capabilities of VFMs for downstream tasks? To answer the above questions, this paper redefines the traditional DG paradigm by introducing the concept of Set Pivot Learning (SPL). SPL aims to enhance model robustness by leveraging the inherent generalization capabilities of VFMs, ultimately improving performance on downstream tasks. Our proposed SPL is characterized by two primary features. Firstly, it employs dynamic adaptation, transitioning from static domain alignment to flexible, task-driven feature optimization, thereby enabling the model to evolve with downstream scenarios. Secondly, SPL utilizes vision foundation models as pivots, leveraging pre-trained knowledge to enhance task-specific representations while preserving cross-domain robustness. Unlike traditional DG that imposes strict domain definitions, SPL prioritizes practical application, moving away from strict domain boundaries. In SPL, accessible training data, such as synthetic or public datasets, are collectively referred to as the Knowledge Set , while the intended application data forms the Application Set (as shown in Fig. 2 ). Furthermore, we investigate challenges emerging within this framework. As shown in Table I , our experiments reveal that even without fine-tuning, frozen VFMs can significantly outperform previous generalization methods. This finding not only validates the inherent generalization capabilities of VFMs, but also shifts the core challenge from traditional model improvement or data augmentation toward strategically magnifying VFMs’ advantages for downstream tasks. Fig. 2: We introduce the concept of Set Pivot Learning , with the goal of leveraging the vision foundation model (VFM) as the backbone to improve the robustness and performance of the model in downstream tasks. To tackle the mentioned challenge, we propose a novel dynamic prompt fine-tuning method that generates prompts from the knowledge set to effectively transfer VFMs’ generalization capabilities to task-specific optimization. Specifically, we design a Dynamic Class-aware Prompter (DCP) to flexibly extract scene-specific prompts from images through category filtering and hierarchical clustering. In addition, we propose a Prompt-guided Feature Focuser (PFF) integrated within the VFM layers, which utilizes similarity prompts, class prompts and a multimodal attention mechanism to fine-tune the VFM. Such a selective fine-tuning strategy enhances model robustness and performance for generalized segmentation. The main contributions of this work can be summarized as follows: • We redefine the domain generalization task and introduce Set Pivot Learning , which aims to leverage the pre-trained knowledge of VFMs as pivot to dynamically adapt and enhance the robustness of the downstream tasks. • We propose a dynamic prompt fine-tuning method with a dynamic class-aware prompter and prompt-guided feature focuser to effectively adapt the generalization strengths of VFMs to specific tasks. • Extensive experiments are conducted to demonstrate the rational of our novel framework, and reveal its superior performance in generalized street scene segmentation. II Related Work II-A Domain Generalization To address the challenge of performance degradation when models encounter data from unseen domains, domain generalization (DG) has emerged as a key strategy. DG methods [ 10 , 11 , 10144687 ] are fundamentally designed to enhance the generalization of models under diverse or unknown distribution conditions. The field of DG can be broadly categorized into two main branches: multiple-source DG and single-source DG. In multiple-source DG [ 12 , 13 , 14 , 15 ] , the model is trained on multiple source domains and then encourage the network to learn domain-invariant feature representations over diverse distributions. Feature alignment-based approaches [ 16 , 17 ] have gained popularity in multiple-source DG, where distribution constraints are enforced