Through the Looking Glass: A Dual Perspective on Weakly-Supervised Few-Shot Segmentation
Jiaqi Ma, Guo-Sen Xie, Fang Zhao, Zechao Li · 2025
Read Through the Looking Glass: A Dual Perspective on Weakly-Supervised Few-Shot Segmentation by Jiaqi Ma, Guo-Sen Xie, Fang Zhao, Zechao Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16159.txt.
Introduction
Jiaqi Ma, Guo-Sen Xie, Fang Zhao, and Zechao Li redesign few-shot image segmentation so the example images and the query image are treated as two different perspectives rather than twins. Meta-learning usually feeds support and query through identical networks, which over-homogenizes what they learn. Some differences between the views carry the real signal.
Their network keeps the two paths homologous in structure but heterogeneous in processing. Added modules boost what is complementary, strip semantic noise, and fold in text features from a vision-language model. The target task is weakly supervised: only cheap labels guide training. On benchmarks the dual view segments new categories from a handful of examples more faithfully.
Catalog record
| CREATOR | Jiaqi Ma, Guo-Sen Xie, Fang Zhao, Zechao Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16159
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Through the Looking Glass: A Dual Perspective on Weakly-Supervised Few-Shot Segmentation Jiaqi Ma, Guo-Sen Xie, Fang Zhao, Zechao Li Abstract Meta-learning aims to uniformly sample homogeneous support-query pairs, characterized by the same categories and similar attributes, and extract useful inductive biases through identical network architectures. However, this identical network design results in over-semantic homogenization. To address this, we propose a novel homologous but heterogeneous network. By treating support-query pairs as dual perspectives, we introduce heterogeneous visual aggregation (HA) modules to enhance complementarity while preserving semantic commonality. To further reduce semantic noise and amplify the uniqueness of heterogeneous semantics, we design a heterogeneous transfer (HT) module. Finally, we propose heterogeneous CLIP (HC) textual information to enhance the generalization capability of multimodal models. In the weakly-supervised few-shot semantic segmentation (WFSS) task, with only 1/24 of the parameters of existing state-of-the-art models, TLG achieves a 13.2% improvement on Pascal-5 i and a 9.7% improvement on COCO-20 i . To the best of our knowledge, TLG is also the first weakly supervised (image-level) model that outperforms fully supervised (pixel-level) models under the same backbone architectures. The code is available at https://github.com/jarch-ma/TLG. I Introduction Fig. 1: (a) Traditional homogeneous network. Support-query pairs of the same category extract similar features and use identical architectures (Projection in the same semantic mapping space), resulting in over-homogenization of semantic features and limiting model performance. (b) Heterogeneous network. Support-query pairs extract distinct features and adopt different design patterns (Projection in the different semantic mapping space), enhancing semantic richness from multiple perspectives and promoting heterogeneous complementarity to improve model performance. 0 0 footnotetext: In data collection, homogeneity refers to the same category and similar attributes (support-query pairs). While in network design, it implies identical architectures for these pairs. The Meta-learning, owing to its strong adaptability, has attracted considerable attention for effectively addressing challenges such as data scarcity and long-tailed distributions in real-world tasks [ 1 , 2 , 3 ] . The field has been extensively studied, leading to the emergence of various methods, such as similarity-based metric learning and prototype-based class-agnostic induction approaches [ 4 , 5 , 6 , 7 ] . However, these methods fail to fully exploit the potential of meta-learning, as they employ identical network architectures for both the support and query branches, thereby overlooking their inherent semantic complementarity. Inspired by the concept of homologous but heterogeneous traits in biological evolution [ 8 ] , we observe that the meta-learning paradigm inherently exhibits homogeneity. Specifically, during data collection, it employs support-query pairs sharing same categories and similar attributes. This process parallels the role of homology in biological evolution, where structural or genetic similarities drive the adaptation of specific functions. Therefore, compared to traditional neural networks, meta-learning inherently offers greater potential to capture richer semantics of target categories. However, as with the aforementioned approaches, current mainstream meta-learning methods have largely overlooked this advantage. As shown in Figure 1 (a), current meta-learning methods still adopt the same training paradigm as traditional neural networks for support-query homologous pairs, utilizing identical network architectures and semantic embedding spaces. This leads the model to overemphasize shared attributes of target categories, such as distinctive ‘head’ features, thereby reinforcing over-homogenization. To bridge this gap, as illustrated in Figure 1 (b), we treat the support and query branches as mutually mapped, complementary semantic embedding spaces. T hrough the L ooking G lass, 1 1 1 Inspired by the classic literary work [ 9 ] , where mirrored and real worlds oppose yet reflect each other, the metaphor parallels TLG’s design: heterogeneous branches sharing a homologous category. we propose a novel homogeneous but heterogeneous network based on a dual-perspective approach, named TLG . By mapping support-query pairs through distinct heterogeneous embedding spaces, TLG captures richer semantic features, unlocking the full potential of meta-learning models. As shown in Figure 2 , TLG consists of three parts: (1) Heterogeneous Aggregation (HA) module. Support and query obtain semantic embeddings from the same but slightly different layers of the backbone, respectively, to enhance heterogeneous complementarity. The correlations between their semantic embedding spaces are then captured, and heterogeneous aggregation maximizes the richness of heterogeneous semantics while preserving semantic commonality. (2) Heterogeneous Transfer module. Due to the aggregation of diverse heterogeneous information, noise is inevitably introduced into the semantic embeddings, affecting the accurate induction of target feature biases. The Heterogeneous Transfer (HT) module can optimize the feature distribution in the semantic embedding space to reduce interference. (3) Heterogeneous clip (HC) module. Purely visual heterogeneous information significantly enhances the semantic representation ability of meta-learning models, yet the generalization capability remains limited. Therefore, we leverage the prior knowledge from CLIP’s textual information to augment TLG’s multimodal representation with heterogeneous textual information, thereby improving the model’s generalization performance. Our insight can be encapsulated in the phrase: Segmentation of the heterogeneous, by the heterogeneous, and for the heterogeneous. Of refers to the segmentation of heterogeneous elements, defined by their diversity and complementarity. By indicates that the segmentation process is driven by heterogeneous feature embeddings. For signifies that the heterogeneous is not only used in segmentation but also serves as a novel design paradigm, advancing the development of meta-learning. To sum up, our contributions can be listed as: • We have identified the issue of homogeneity in the meta-learning paradigm and propose a novel homogenous but heterogeneous network design paradigm. • To validate the effectiveness of the homogenous but heterogeneous network, we propose a multimodal-based visual Heterogeneous Aggregation (HA) Module and Heterogeneous Transport (HT) Module, as well as a Heterogeneous CLIP (HC) Module based on Clip textual information. • In the WFSS task, with only 1/24 of the parameters of existing state-of-the-art models, TLG achieves a 13.2% improvement on Pascal-5 i and a 7.9% improvement on COCO-20 i . • To the best of our knowledge, TLG is the first weakly-supervised model (image-level labels) with the same backbone to achieve superior performance compared to existing fully-supervised (pixel-level labels) state-of-the-art models. Fig. 2: Overview of our TLG Framework. (1) Heterogeneous Aggregation (HA) module. To complement heterogeneous semantics based on shared semantics, intra-layer heterogeneity is leveraged (e.g., using layer 3 for support and layer 0 for query within the lower layers 0–3), along with inter-layer correlations. (2) Heterogeneous Transport (HT) module. The HA module enhances contextual relevance through attention transport, while suppressing noise via optimal transport, and subsequently enriches semantic diversity with heterogeneous residuals. (3) Heterogeneous CLIP (HC) module. By incorporating multimodal heterogeneous CLIP text information, the robustness of TLG is enhanced. II Related Workss II-A Meta Learning Traditional neural networks assume that training and test data are independent and identically distributed (i.i.d.) [ 10 , 11 , 12 ] . However, this assumption often deviates from real-world scenarios, leading to significant performance degradation when models encounter out-of-distribution domains [ 13 ] . Meta-learning, or learning-to-learn, aims to improve the learning algorithm itself by leveraging knowledge from similar or related tasks, thus enabling generalization across multiple domains [ 1 , 14 , 15 , 16 ] . This paradigm offers new solutions to address current challenges related to data and computational bottlenecks, as well as generalization issues. However, current meta-learning is still limited by homogenized network design strategies, which constrain the potential of model performance [ 17 , 18