Visual Adaptive Prompting for Compositional Zero-Shot Learning

Kyle Stein, Andrew Arash Mahyari, Guillermo Francia III, Eman El-Sheikh · 2025

Read Visual Adaptive Prompting for Compositional Zero-Shot Learning by Kyle Stein, Andrew Arash Mahyari, Guillermo Francia III, Eman El-Sheikh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20292.txt.

Introduction

Kyle Stein, Arash Mahyari, Guillermo Francia, and Eman El-Sheikh present a prompting system that lets vision-language models name attribute-object pairs they never saw combined in training. Their paper appeared in two thousand twenty-five. Recognizing a red car is easy if red things and cars were seen apart. Static text prompts, though, ignore what the image actually shows.

Their system keeps a library of learnable visual prompts for attributes and objects. For each image, it retrieves the prompts that best match the visual features. An adapter steers the model toward an embedding space that generalizes. Tests on three benchmarks, in closed and open settings, set new best results. The authors argue composition needs vision-led adaptation, not text alone. The method suits any task where familiar parts recombine. It needs no retraining of the base model.

Catalog record

CREATORKyle Stein, Andrew Arash Mahyari, Guillermo Francia III, Eman El-Sheikh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-20292

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visual Adaptive Prompting for Compositional Zero-Shot Learning Kyle Stein, Andrew Arash Mahyari, Guillermo Francia III, Eman El-Sheikh Abstract Vision-Language Models (VLMs) have demonstrated impressive multimodal capabilities in learning joint representations of visual and textual data, making them powerful tools for tasks such as Compositional Zero-Shot Learning (CZSL). CZSL requires models to generalize to novel combinations of visual primitives—such as attributes and objects—that were not explicitly encountered during training. Recent works in prompting for CZSL have focused on modifying inputs for the text encoder, often using static prompts that do not change across varying visual contexts. However, these approaches struggle to fully capture varying visual contexts, as they focus on text adaptation rather than leveraging visual features for compositional reasoning. To address this, we propose a V isual A daptive P rompting S ystem (VAPS) that leverages a learnable visual prompt repository and similarity-based retrieval mechanism within the framework of VLMs to bridge the gap between semantic and visual features. Our method introduces a dynamic visual prompt repository mechanism that selects the most relevant attribute and object prompts based on the visual features of the image. Our proposed system includes a visual prompt adapter that encourages the model to learn a more generalizable embedding space. Experiments on three CZSL benchmarks, across both closed and open-world scenarios, demonstrate state-of-the-art results. 1 Introduction Humans have a remarkable ability to compose attributes with objects to imagine novel combinations they have never encountered, for example, a blue banana. Attributes describe the state of an object, such as the color, texture, or shape, while objects represent the entities themselves, such as a banana or a car. This ability to associate various attributes with different objects is a fundamental aspect of human cognition, known as compositionality [ 15 , 16 , 4 ] . Compositional Zero Shot Learning (CZSL) aims to mimic this human behavior by enabling models to recognize combinations of primitive attributes and objects that were not explicitly composed, or seen together, during training. CZSL focuses on the recombination of known primitives, allowing for the recognition of novel compositions by effectively disentangling attribute and object information from a combined visual representation [ 5 , 7 , 39 ] . Figure 1 : Similarity-Based Visual Repository Mechanism. The image features are matched against learned keys in a shared repository space, where similarity scores are computed. Based on these similarity scores, the most relevant learned visual prompts representing attributes (i.e., “color”) and objects (i.e., “animals”) are selected. The selected prompts are then averaged, creating a final representation (purple) that holds the object and attribute information of the input images. This process allows dynamic retrieval and combination of relevant visual prompts for downstream tasks. The rigorous pre-training of VLMs have shown great ability for understanding tasks that require multimodal visual and textual data. Models like CLIP [ 32 ] are trained on vast amounts of image-text data and learn to map visual and textual representations into a shared embedding space, enabling them to perform tasks such as zero-shot classification and image recognition. However, when approached with the more nuanced understanding of compositional reasoning, CLIP lacks the ability to generalize to unseen primitive compositions. This lack of generalization can be attributed to the reliance on static representations of fixed classes seen during training and the lack of flexibility to adjust their internal prompts to changes in visual inputs [ 9 , 10 , 42 ] . Recent studies in CZSL based on VLMs [ 38 , 1 , 22 ] suffer from several drawbacks which limit their performance. First , state-of-the-art (SOTA) methods typically utilize either fixed templates “ a photo of [attribute][object] ” or a single learned prefix prepended to “ [attribute][object] ”. In these strategies, the text prompt is processed through the text encoder, while the raw image is handled by the image encoder, leading to the development of a joint embedding that facilitates the inference of unseen attribute-object compositions. Although promising, text-focused methods largely overlook valuable visual insights, as they concentrate on tuning text-based prompts rather than exploiting visual information that could enhance attribute-object disentanglement and improve model adaptability to unseen compositions [ 23 , 26 ] . Second , SOTA algorithms typically rely solely on these text-centric learnable prompts, and operate under the assumption that a minimal number of prompts can adequately capture all attribute-object combinations. For example, a single prompt might be used to denote a variety of attributes like ”wet,” ”dry,” ”red,” etc. This limited approach constrains the development of tailored prompts that could significantly improve performance. Moreover, using only one or two prompts means these techniques struggle to properly separate attribute features from object features, thereby restricting their effectiveness in generalizing to unseen compositions. Third , text-centric approaches utilize fixed prompts as a prefix to “ [attribute][object] ” during training. These static prompts are comprised of a set of learnable variables that remain constant in their positions across different attribute-object pairings, such as transitioning from “wet cat” to “red apple”. Relying on such static, learnable text prompts often fails to fully encompass the entire context of an image. This is because text descriptions can be relatively rigid, not adequately reflecting the complexities of an image. For instance, the attribute “wet” might carry different semantic implications when associated with disparate objects like a cat versus an apple. To address these challenges, we propose a novel approach Visual Adaptive Prompting System (VAPS) that builds on CLIP’s multi-modal architecture by leveraging a dynamic visual prompt repository and a similarity-based retrieval mechanism, which shifts the emphasis to the image features generated by CLIP’s visual encoder. VAPS creates a repository of visual features to serve as visual prompts, comparing them with fused features in a pair space, as well as implementing a prompt adapter based on the original image features that allows the model to adapt its representation based on the visual context. The summary of our contributions for CZSL are stated below: • Visual Prompts: To leverage information directly from the image encoder, we introduce visual prompts. These visual prompts are learnable embeddings designed to capture visual patterns related to attributes and objects. This high-level semantic representation efficiently separates attributes from object visual features, allowing VAPS to generalize visual semantics to unseen compositions more effectively. Unlike text-based prompts, visual prompts leverage visual features to enhance disentanglement and boost adaptability. • Prompts Repository: Our approach employs a repository of learnable visual prompts that operate independently from those used by the text encoder. Each visual prompt in this repository is paired with a learnable key, which serves as an identifier for effective selection. VAPS uses a similarity-based retrieval mechanism to match image features with learned keys in the repository, selecting the most relevant prompts for the input image. This allows the model to effectively disentangle attributes from objects, facilitating generalization to unseen compositions. • Text Prompt Adapter: VAPS incorporates an adapter that dynamically updates the prefix of the text prompt using image features from the visual transformer. By incorporating a bias term from image features, this approach customizes the prompt for each image, aligning it with the visual context and effectively separating attributes from objects. For instance, the learnable text prompts are adjusted with different bias values when

Read the complete text