Feature Projection Learning for Better Vision-Language Reasoning

Yi Zhang, Weicheng Lin, Liang-Jie Zhang \sthanks Corresponding author · 2026

Read Feature Projection Learning for Better Vision-Language Reasoning by Yi Zhang, Weicheng Lin, Liang-Jie Zhang \sthanks Corresponding author (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20224.txt.

Introduction

Yi Zhang, Weicheng Lin, and Liang-Jie Zhang study how to adapt vision-language models like CLIP to new tasks with little labeled data. CLIP learns from images and text together and extracts features that transfer well. The challenge is to reuse that knowledge downstream without full retraining.

The authors propose feature projection learning as the adaptation route. Better light-weight adaptation would let small teams use strong pretrained models on their own tasks. It also cuts the compute and data needed to reach good accuracy, which widens access to modern vision-language tools.

Catalog record

CREATORYi Zhang, Weicheng Lin, Liang-Jie Zhang \sthanks Corresponding author
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20224

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Feature Projection Learning for Better Vision-Language Reasoning Yi Zhang, Weicheng Lin, Liang-Jie Zhang \sthanks Corresponding author Abstract Vision-Language Pre-Trained models, notably CLIP, that utilize contrastive learning have proven highly adept at extracting generalizable visual features. To inherit the well-learned knowledge of VLP models for downstream tasks, several approaches aim to adapt them efficiently with limited supervision. However, these methods either suffer from limited performance, excessive learnable parameters, or extended training times, all of which hinder their effectiveness in adapting the CLIP model to downstream tasks. In this work, we propose a simple yet efficient and effective method called F eature P rojection L earning(FPL) to address these problems. Specifically, we develop a projection model that projects class prototype features into the query image feature space and reconstructs the query image feature map. The negative average squared reconstruction error is used as the class score. In this way, we transform the classification problem into a feature projection problem. The final output of this method is a combination of the prediction from the projection model and the original pre-trained CLIP. Comprehensive empirical evaluations confirm that FPL delivers superior accuracy, surpassing the current state-of-the-art methods by a substantial margin. † † address: College of Computer Science and Software Engineering, Shenzhen University, China 1 Introduction Foundational models have advanced significantly due to emerging studies on pre-trained Vision-Language Models (VLMs) such as CLIP [ 17 ] , which have demonstrated outstanding performance on open-vocabulary tasks. By harnessing visual semantics derived from massive collections of image-text pairs, these architectures exhibit remarkable proficiency across a wide range of downstream applications, often with zero-shot or few-shot learning [ 17 , 1 ] . Nevertheless, while zero-shot CLIP yields competitive results across general visual benchmarks, its reliance on pre-training often limits its ability to generalize effectively to specific, unencountered domains. Therefore, several works focus on fine-tuning these pre-trained VLMs for downstream tasks through designing learnable prompts derived from training instances. These fine-tuning methods can be categorized into input-stage prompting such as CoOp [ 26 ] , CoCoOp [ 25 ] and feature-stage fine-tuning such as TaskRes [ 23 ] , CLIP-Adapter [ 7 ] , and Tip-Adapter [ 24 ] . Specifically, CoOp [ 26 ] pioneered the use of a learnable prompt to capture task-specific information. This approach was refined by CoCoOp [ 25 ] , which dynamically adjusts prompts on individual image instances. Furthermore, TaskRes [ 23 ] introduces a decoupled task residual to ensure CLIP’s pre-existing knowledge remains intact. On the adapter front, CLIP-Adapter [ 7 ] employs an extra feature layer to enhance fine-tuning. Building on this, Tip-Adapter [ 24 ] offers a training-free strategy by constructing a key-value cache from few-shot data; this cache can subsequently serve as a robust initialization for fine-tuning the keys (Tip-Adapter-F) to improve classification precision. However, these methods either suffer from limited performance, excessive learnable parameters, or extended training times, all of which hinder their effectiveness in adapting the CLIP model to downstream tasks. For example, CoOp requires a substantial amount of training time but exhibits limited few-shot accuracy, While Tip-Adapter-F yields better performance, its large cache size introduces enormous learnable parameters. We then raise the question: Is it possible to improve the model’s performance without introducing additional parameters or increasing the training time? To address the problems above, in this work, we propose a parameter-efficient method called Feature Projection Learning (FPL) to address these problems. We convert the classification problem into a projection problem. Specifically, given a set of images from the same class, we produce the associated feature maps and collect the component feature vectors across locations and images into a single pool of support features, we name it class prototype features. For each query image, We then attempt to project prototype features of each class into the query image feature space, and reconstruct the query image feature map as a weighted sum of class prototype features. The negative average squared reconstruction error is used as the class score. Intuitively, reconstructing images from the same class should be simpler because they share similar embeddings. Conversely, images from different classes will be more challenging to reconstruct and may result in large reconstruction errors. Assessing the reconstruction of the entire query image feature map enables us to retain the spatial details it contains. Additionally, permitting this reconstruction to utilize feature vectors from any location in the support images intentionally eliminates irrelevant location-specific information. Innovatively, we approach feature map projection as a ridge regression task, which enables us to efficiently compute a solution using a straightforward closed-form approach, employing only a single learned, soft constraint. The final output of our proposed FPL is a combination of prediction from the projection model and original pre-trained CLIP. Our extensive experimental results on few-shot classification and domain generalization demonstrate that our proposed FPL method can significantly improve CLIP’s adaptation capability, and outperforms existing state-of-the-art methods, for example, by up to 5.1% improvement on few-shot learning and by up to 4.3% for domain generalization. 2 Method Figure 1: Overview of our FPL method . First, we generate the feature map of the query image by E_{vm} , for each class, we pool the features generated from the set of few-shot available images by E_{vm} into a unified feature matrix, then we use projector to project these feature matrix into query image feature space and reconstruct the query image feature map. We calculate the negative squared Euclidean distance between the reconstructed query feature map and the real query feature map as the classification score. Combined with the prediction of the original CLIP, we got the final prediction. Note that E_{vm} is the modified image encoder of CLIP that does not include the last attention pooling layer. 2.1 A Revisit of CLIP CLIP [ 17 ] is a pioneering VLP model that consists of two parallel encoders for both textual and visual modalities. We denote these two encoders as E_{v} and E_{t} respectively. The two encoders of CLIP will map the text descriptions and images into the same embedding space. In the training stage, CLIP adopts a contrastive loss to align two modalities. In this work, we consider the problem of using CLIP for a K -classes image classification problem. For this specific setting, in the textual branch, we append a user-defined prompt such as \sigma= “a photo of …” with each class name y_{i} in the set Y=\{y_{1},y_{2},\cdots,y_{K}\} to get class-specific text descriptions. Using two encoders E_{v} and E_{t} , we can generate the image feature f_{v} and K text feature f_{t_{i}} , which corresponds to class name y_{i} . Based on the cosine similarity of these features, we can make the prediction of the test image X_{test} by p(y_{i}|X_{test})=\frac{\exp\left(\mathrm{sim}\left(f_{t_{i}},f_{v}\right)/\tau\right)}{\sum\nolimits_{j=1}^{K}{\exp\left(\mathrm{sim}\left(t_{j},v\right)/\tau\right)}}, (1) where \tau is the temperature hyper-parameter and \mathrm{sim}(\cdot,\cdot) denotes the following cosine similarity: \mathrm{sim}\left(f_{t_{i}},f_{v}\right)=\frac{f_{t_{i}}\cdot f_{v}}{\|f_{t_{i}}\|\|f_{v}\|}. (2) 2.2 Prototype-to-Query Projection Model As shown in Figure 1 , our model is built upon CLIP, using E_{t} as the text encoder and E_{v} as the image encoder. To begin with, we introduce the the architecture of visual encoder in detail. Take the ResNet encoder for example, CLIP makes a small modification by appending an attention pooling layer to the conventional ResNet. We denote the encoder without attention

Read the complete text