SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning
Zhi Chen, Zecheng Zhao, Jingcai Guo, Jingjing Li, Zi Huang · 2025
Read SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning by Zhi Chen, Zecheng Zhao, Jingcai Guo, Jingjing Li, Zi Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10252.txt.
Introduction
Zhi Chen, Zecheng Zhao, Jingcai Guo, Jingjing Li, and Zi Huang tackled zero-shot learning with cleaned image patches in 2025. Zero-shot systems name classes they never saw in training, guided by class descriptions. A core flaw is misalignment: image patches with no link to the class text pollute the match.
Rivals clean features after the fact; this method acts at the input. A self-taught selector scores each patch using attention pooled from all layers, then drops the patches judged unrelated. To keep objects whole, dropped patches are swapped for learned stand-ins seeded from word vectors. On standard benchmarks the method set new best scores. Its features are also easier to read and richer in meaning, the authors say.
Catalog record
| CREATOR | Zhi Chen, Zecheng Zhao, Jingcai Guo, Jingjing Li, Zi Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10252
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning Zhi Chen, Zecheng Zhao, Jingcai Guo, Jingjing Li, Zi Huang Abstract Zero-shot learning (ZSL) aims to recognize unseen classes without labeled training examples by leveraging class-level semantic descriptors such as attributes. A fundamental challenge in ZSL is semantic misalignment, where semantic-unrelated information involved in visual features introduce ambiguity to visual-semantic interaction. Unlike existing methods that suppress semantic-unrelated information post hoc either in the feature space or the model space, we propose addressing this issue at the input stage, preventing semantic-unrelated patches from propagating through the network. To this end, we introduce S emantically contextualized VI sual P atches ( SVIP ) for ZSL, a transformer-based framework designed to enhance visual-semantic alignment. Specifically, we propose a self-supervised patch selection mechanism that preemptively learns to identify semantic-unrelated patches in the input space. This is trained with the supervision from aggregated attention scores across all transformer layers, which estimate each patch’s semantic score. As removing semantic-unrelated patches from the input sequence may disrupt object structure, we replace them with learnable patch embeddings. With initialization from word embeddings, we can ensure they remain semantically meaningful throughout feature extraction. Extensive experiments on ZSL benchmarks demonstrate that SVIP achieves state-of-the-art performance results while providing more interpretable and semantically rich feature representations. 1 Introduction Figure 1 : Approaches to mitigate semantic misalignment for ZSL. (a) Disentangling methods remove semantic-unrelated information in the feature space. (b) Progressive semantic-guided methods remove the semantic-unrelated patches in the model space. (c) Our SVIP identifies the semantic-unrelated image patches in the input place and replace them with semantically contextualized visual patches . Zero-shot learning (ZSL) [ 33 , 47 , 23 ] is a fundamental learning paradigm in computer vision that enables models to recognize classes unseen during training. ZSL methods typically recast classification as a regression task, where class-level semantic descriptors, such as attributes [ 18 ] serve as soft labels. For instance, each dimension of an attribute vector may indicate the presence or absence of a concept like the color pink. At test time, the predicted attribute vector is compared against the attributes of unseen classes, and the most similar class is selected. Semantic misalignment is a fundamental challenge in ZSL [ 42 , 22 , 1 , 8 , 4 , 26 ] . Raw images often contain semantic-unrelated details that are not described by attributes. For example, background clutter, lighting variations, or surrounding objects can introduce misleading patterns. As a result, the model may latch onto unimportant features instead of the relevant semantic cues. Since these extraneous details are not explicitly annotated in attribute vectors, they disrupt the visual-semantic alignment by diluting key attributes such as color and shape. This makes it harder for the model to recognize unseen classes. As shown in Figure 1 (a), prior work attempted to address this issue by refining or disentangling features, so that semantic-consistent information is preserved while semantic-unrelated information is suppressed [ 8 , 1 , 22 , 26 ] . Thus, the preserved semantic-consistent visual features can enhance visual-semantic alignment. Although these post hoc refinements help, they cannot fully prevent extraneous information from seeping into the learned representations. In parallel, transformer-based architectures [ 44 ] have emerged as powerful backbones in vision tasks [ 21 ] by modeling images as sequences of patches. In light of the strong representation learning capability of Vision Transformers (ViTs) [ 16 , 52 ] , researchers in ZSL community are turning into designing ViT-based methods [ 4 , 14 , 27 ] . The patch modelling behavior enables identifying the semantic-unrelated patches. As in Figure 1 (b), ZSLViT [ 4 ] progressively prunes visual tokens that do not map to semantics in the model space. However, research has shown that semantic features can become diluted in deeper layers of ViTs [ 36 ] , making it easier for semantic-unrelated information to persist and potentially contaminate the final visual features. As above, these methods either remove the semantic-unrelated features after extracting the entangled visual features, or during extracting the visual features, they do not entirely prevent such features from influencing learned representations. Consequently, an intriguing research question arises: Can we preemptively remove or repurpose semantic-unrelated patches at the input stage to avoid entangling them in the learned features? To achieve this goal, we propose SVIP , S emantically contextualized VI sual P atch, a ViT-based framework tailored for mitigating the semantic-unrelated information and enhance semantic-visual propagation in ZSL. As shown in Figure 1 (c), unlike prior work that prunes or refines features after they have been extracted, SVIP identifies and contextualizes these patches upfront. Specifically, to mitigate the semantic misalignment problem comprehensively, we propose self-supervised patch selection mechanism that learns to identify the semantic-unrelated patches in the input space, i.e., before feeding into the transformer blocks. In particular, we aggregate the self-attention weight matrices across all transformer blocks, yielding reliable semantic understanding about the patch importance throughout the entire ViT. To mitigate the negative impact of semantic-unrelated patches to semantic-related patches, we train a patch classifier to determine the input patches that are semantic-unrelated patches, using the supervision from the aggregated attention matrix. At the test time, the patch classifier can directly prevent the semantic-unrelated patches from propagating into the feature extraction. Once the semantic-unrelated patches are identified in the input sequence, a naive solution is removing them from the input sequence. However, this can break object structure on the image. Instead, we propose to mask the semantic-unrelated patches with a learnable semantic context patch. The semantic context patch is initialized with the attribute-level word embeddings that provides rich semantic information. These contextualize input patches can enhance the visual-semantic interaction throughout the visual feature extraction process. Overall, the contributions of this paper are: • We introduce S emantically contextualized VI sual P atch ( SVIP ), a ViT-based framework for addressing the semantic misalignment problem in zero-shot learning. It emphasizes the early-stage handling of semantic-unrelated patches to achieve a more focused and unambiguous feature space for ZSL. • We propose a self-supervised patch selection strategy to identify semantic-unrelated patches using the attention scores aggregated from the self-attention weight matrix across the ViT blocks. We propose patch semantic contextualization that injects of attribute-level word embeddings into semantic-unrelated patches, ensuring that semantic information is incorporated into visual representations from the initial layers of the transformer. • SVIP achieves state-of-the-art performance results on three ZSL benchmark datasets on both ZSL and generalized ZSL settings, demonstrating the effectiveness in mitigating semantic misalignment in the input space. 2 Related Work The core objective in ZSL [ 38 , 47 , 23 , 49 , 8 , 6 , 41 , 7 , 13 , 20 , 19 , 10 ] is to generalize visual-semantic knowledge obtained from seen classes to unseen classes. This generalization relies on semantic descriptors, which are typically represented as attributes [ 18 ] , documents [ 17 ] , or word embeddings [ 32 ] . The main stream ZSL methods can be generally categorized as generative methods and embedding-based methods. Generative ZSL methods typically synthesize unseen visual features from semantic descriptors using generative models, enabling standard classification pipelines to incorporate synthetic training samples [ 49 ,