Unify the Views: View-Consistent Prototype Learning for Few-Shot Segmentation

Hongli Liu, Yu Wang, Shengjie Zhao, [0.6em] School of Computer Science, Technology, Tongji University[0.4em] [email protected], [email protected], [email protected] [0.2em] · 2026

Read Unify the Views: View-Consistent Prototype Learning for Few-Shot Segmentation by Hongli Liu, Yu Wang, Shengjie Zhao, [0.6em] School of Computer Science, Technology, Tongji University[0.4em] [email protected], [email protected], [email protected] [0.2em] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05952.txt.

Introduction

Liu and team present VINE for few-shot splitting under mis-match and cross-view clash. Few-shot splitting spreads to novel classes with limited guidance, yet struggles across views. VINE models shape sameness and foreground telling apart, tuning models. A space-view graph captures local layout and spreads view-fixed meaning. A telling prior from support-query gap shifts SAM features toward key regions. Enhanced SAM and ResNet traits blend through masked cross-focus into class-steady models. Tests back power and toughness under viewpoint shifts and complex structures.

Catalog record

CREATORHongli Liu, Yu Wang, Shengjie Zhao, [0.6em] School of Computer Science, Technology, Tongji University[0.4em] [email protected], [email protected], [email protected] [0.2em]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37167
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05952

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unify the Views: View-Consistent Prototype Learning for Few-Shot Segmentation Hongli Liu, Yu Wang, Shengjie Zhao, [0.6em] School of Computer Science, Technology, Tongji University[0.4em] [email protected], [email protected], [email protected] [0.2em] Abstract Few-shot segmentation (FSS) has gained significant attention for its ability to generalize to novel classes with limited supervision, yet remains challenged by structural misalignment and cross-view inconsistency under large appearance or viewpoint variations. This paper tackles these challenges by introducing VINE (View-Informed NEtwork), a unified framework that jointly models structural consistency and foreground discrimination to refine class-specific prototypes. Specifically, VINE introduces a spatial–view graph on backbone features, where the spatial graph captures local geometric topology and the view graph connects features from different perspectives to propagate view-invariant structural semantics. To further alleviate foreground ambiguity, we derive a discriminative prior from the support-query feature discrepancy to capture category-specific contrast, which reweights SAM features by emphasizing salient regions and recalibrates backbone activations for improved structural focus. The foreground-enhanced SAM features and structurally enriched ResNet features are progressively integrated through masked cross-attention, yielding class-consistent prototypes used as adaptive prompts for the SAM decoder to generate accurate masks. Extensive experiments on multiple FSS benchmarks validate the effectiveness and robustness of VINE, particularly under challenging scenarios with viewpoint shifts and complex structures. The code is available at https://github.com/HongliLiu1/VINE-main . 1 Introduction Semantic segmentation assigns category-level labels to every pixel and plays a foundational role in visual scene understanding 16 ; 26 ; 3 . Nevertheless, fully-supervised segmentation methods rely on large-scale pixel-wise annotations, which are costly to obtain and difficult to scale, particularly for long-tailed or open-set scenarios. Few-Shot Segmentation (FSS) 24 ; 15 ; 22 ; 34 ; 29 addresses this limitation by enabling segmentation of novel categories with only a few labeled examples. By leveraging support-query interactions to transfer knowledge from base to novel classes, FSS establishes a label-efficient paradigm that generalizes beyond annotated categories. Figure 1 : Comparison between conventional prototype modeling and VINE. (a) View-induced challenges: large intra-class variation and inter-class similarity lead to prototype confusion. (b) Top: conventional mask-guided enhancement fails under viewpoint shifts, generating unstable prototypes. Bottom: VINE unifies spatial–view graph alignment and discriminative modulation to produce view-consistent, structurally reliable prototypes. Most few-shot segmentation (FSS) methods follow a prototype paradigm: class representations are distilled from support features and matched against the query to drive dense prediction. Early systems 33 ; 28 form prototypes via global average pooling and segment by feature similarity. Subsequent work replaces global pooling with cross-attention 1 ; 25 ; 18 ; 34 to model pixel-level support–query correspondences and inject support cues into the query stream. Despite these advances, Fig. 1 (a) reveals a persistent failure mode: cross-view geometric variation and fine-grained inter-class resemblance (e.g., “Cat” vs. “Cow”) often destabilize correspondence, leading to prototype drift and boundary ambiguity. In essence, structural consistency, which ensures part–whole geometric preservation across viewpoints, remains insufficiently constrained. Recent pipelines have turned to foundation models, notably SAM 8 ; 39 , which offer prompt-driven mask generation as a source of generalizable cues. Despite these advances, as illustrated in Fig. 1 (b), SAM’s instance-centric saliency prior and sensitivity to prompt placement do not transfer reliably in FSS, where support and query depict different instances with substantial pose and shape variation. As a result, pseudo-mask guidance often propagates inconsistently across mismatched poses (e.g., lateral to frontal views), imposing incorrect structural cues, while the lack of explicit foreground–background modeling makes the system vulnerable to noisy masks 36 and occlusion. These observations naturally raise a key question: how can we explicitly couple cross-view structural alignment with robust, query-adaptive foreground discrimination? To address these challenges, we propose VINE (View-Informed Network), a unified framework that jointly enforces structural consistency and semantic discrimination for robust prototype learning under cross-view variations. We posit that high-quality visual prompts should maintain spatial alignment and class consistency while suppressing background noise. VINE adopts a dual-encoder architecture that combines semantic-rich representations from the frozen SAM encoder and structure-sensitive features from ResNet. It integrates two complementary modules: the Spatial-View Graph Alignment (SVGA) module constructs spatial–view graphs over ResNet features to model intra-class geometry across viewpoints, where a prototype loss further enforces alignment between query features and view-modeled prototypes; the Discriminative Foreground Modulation (DFM) mechanism generates class-consistent priors through prototype–query interaction, enhancing discriminative focus and suppressing distractors. As illustrated in Fig. 1 (b), by decoupling geometry and semantics and guiding query features with appropriate priors, VINE produces in-distribution prototypes with improved cross-view generalization. Finally, learnable tokens encode aggregated foreground and class-consistent cues via cross-attention with SAM and ResNet features, and the fused prototypes serve as prompts for SAM’s decoder to generate final masks. In summary, our contributions are as follows: • We propose VINE, a unified framework that enhances robust few-shot segmentation by jointly modeling structural consistency and foreground discrimination. • We design Spatial-View Graph Alignment to build spatial–view graphs that capture intra-class structure and cross-view consistency, with a prototype loss enforcing prototype-level correspondence across views. • We introduce Discriminative Foreground Modulation, leveraging prototype-query interaction to generate foreground-aware priors for improved discriminability. • We demonstrate the effectiveness of the proposed view-informed framework through extensive experiments on standard few-shot segmentation benchmarks with diverse viewpoints and complex scenes. 2 Related Work 2.1 Metric-Based FSS Few-Shot Segmentation (FSS) aims to segment novel-class objects in query images using only a few labeled support examples. Metric-based approaches address this by extracting class prototypes from support features and matching them with query features in a shared embedding space 24 ; 28 ; 34 ; 32 . Early methods typically compute a single or few global prototypes and perform pixel-wise similarity matching, which often overlooks spatial structure and fails under significant appearance variation. To enhance support–query alignment, attention-based methods 40 ; 38 ; 11 ; 20 ; 1 ; 18 ; 41 incorporate fine-grained interactions by dynamically fusing each query pixel with dense foreground features from the support set to emphasize semantically relevant regions. Recent efforts incorporate structure-aware representations 12 ; 23 to model contextual dependencies via part-level reasoning, instance associations. These methods often overlook intra-class variation and lack mechanisms to enforce cross-view consistency, leading to suboptimal alignment under large appearance and geometric changes. Figure 2 : An overview of our proposed framework. We incorporate Spatial-View Graph Alignment (SVGA) to capture structural consistency across views, and Discriminative Foreground Modulation (DFM) to highlight foreground-relevant features, collaboratively improving the quality of support-query prototypes and guiding accurate mask prediction. 2.2 Foundation Model-Based FSS Recent advances have integrated foundation models, particularly the Segment Anything Model (SAM) 8 , into Few-Shot Segmentation (FSS) to enhance generalization and reduce annotation costs 14 ; 31 ; 7 ; 17 ; 13 . SAM provides strong class-agnostic segmentation via prompt-based guidance, making it suitable for low-supervision settings. Beyond SAM-based pipelines, HMNet 37 employs a hybrid Mamba architecture to strengthen long-range dependency modeling for prototype construction, while UniFSS 2 redefines guidance by unifying cues beyond masks to improve supervision. Existing SAM-enhanced approaches such as VRP-SAM 27 and SAM-RS 35 improve prompt quality through visual-reference encoding and multimodal refinement, and FCP 21 adopts prototype-guided prompting to align support–query semantics. However, these methods largely rely on appearance similarity or prompt refinement, lacking explicit mechanisms to preserve structural coherence and foreground–background separation under geometric variation. Therefore, coupling

Read the complete text