TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning
Jingjing Xie, Yuxin Zhang, Jun Peng, Zhaohong Huang, Liujuan Cao · 2024
Read TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning by Jingjing Xie, Yuxin Zhang, Jun Peng, Zhaohong Huang, Liujuan Cao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08176.txt.
Introduction
Jingjing Xie, Yuxin Zhang, and their co-authors present TextRefiner, a 2024 add on that sharpens prompt tuning for vision language models. Prompt learning adapts such models cheaply, but the learned prompts are usually shared across all classes, so they blur fine visual differences between similar classes. Other fixes borrow descriptions from large language models at some running cost.
TextRefiner instead mines the vision model's own image branch. A local cache module gathers fine visual concepts from image tokens. Those concepts are pooled and aligned with the text branch's output, enriching the prompts from within the model itself. The module plugs into existing prompt methods without outside knowledge. Tests showed better class separation at no added inference burden from external models.
Catalog record
| CREATOR | Jingjing Xie, Yuxin Zhang, Jun Peng, Zhaohong Huang, Liujuan Cao |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08176
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TextRefiner: Internal Visual Feature as Efficient Refiner for Vision-Language Models Prompt Tuning Jingjing Xie, Yuxin Zhang, Jun Peng, Zhaohong Huang, Liujuan Cao Abstract Despite the efficiency of prompt learning in transferring vision-language models (VLMs) to downstream tasks, existing methods mainly learn the prompts in a coarse-grained manner where the learned prompt vectors are shared across all categories. Consequently, the tailored prompts often fail to discern class-specific visual concepts, thereby hindering the transferred performance for classes that share similar or complex visual attributes. Recent advances mitigate this challenge by leveraging external knowledge from Large Language Models (LLMs) to furnish class descriptions, yet incurring notable inference costs. In this paper, we introduce TextRefiner, a plug-and-play method to refine the text prompts of existing methods by leveraging the internal knowledge of VLMs. Particularly, TextRefiner builds a novel local cache module to encapsulate fine-grained visual concepts derived from local tokens within the image branch. By aggregating and aligning the cached visual descriptions with the original output of the text branch, TextRefiner can efficiently refine and enrich the learned prompts from existing methods without relying on any external expertise. For example, it improves the performance of CoOp from 71.66 % to 76.94 % on 11 benchmarks, surpassing CoCoOp which introduces instance-wise features for text prompts. Equipped with TextRefiner, PromptKD achieves state-of-the-art performance and is efficient in inference. Our code is relesed at https://github.com/xjjxmu/TextRefiner . 1 Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Xiamen, China Introduction Vision-Language Models (VLMs), such as CLIP ( Radford et al. 2021 ) and ALIGN ( Jia et al. 2021 ) , have manifested exceptional generalization abilities, significantly benefiting a vast spectrum of applications like open-vocabulary image recognition ( Zhai et al. 2023 ) , object detection ( Gu et al. 2022 ) , and image segmentation ( Li et al. 2022 ) . Generally, VLMs optimize an image encoder and a text encoder to align text and image features in a unified embedding space via contrastive loss. By pre-training on large-scale web data, VLMs can acquire transferable representations in both visual and textual domains, facilitating their application to a broad range of downstream visual tasks. As such, numerous research efforts have been directed towards devising approaches to adapt VLMs in a manner mindful of both parameters and data ( Khattak et al. 2023a ; Shen et al. 2024 ; Gao et al. 2024b ; Yu et al. 2023 ; Li et al. 2024a ) . To date, the predominant focus of research has predominantly fallen on employing prompt learning ( Zhou et al. 2022b ; Zhou et al. 2022a ; Kunananthaseelan, Zhang, and Harandi 2024 ; Khattak et al. 2023b ) , a widely acknowledged strategy for adapting large pre-trained models. As an initial foray, CoOp ( Zhou et al. 2022b ) leveraged prompt learning to adapt text prompts of CLIP, resulting in marked performance improvements in downstream tasks. Subsequent research efforts have sought to extend prompt learning into the visual ( Wu et al. 2023 ; Li et al. 2024b ; Shen et al. 2024 ) and multimodal domains ( Khattak et al. 2023a ; Xing et al. 2023 ; Gao et al. 2024a ; Xin et al. 2024 ) , aiming to thoroughly fine-tune the representations across modalities to enhance their alignment. Figure 1: Comparison of different paradigms for enriching text prompts. Previous methods (left) introduced external knowledge experts to furnish fine-grained descriptions of each class, necessitating an extra filtering process to maintain alignment with downstream datasets. In contrast, our proposed TextRefiner (right) leverages the internal knowledge of the image branch to supply fine-grained, localized region information, thereby drastically reducing the inference overhead while maintaining the performance. Despite the ongoing advancements, a prevalent limitation of these works is their tendency to enhance high-level semantics in a coarse-grained, global manner, resulting in holistic alignment across modalities. Consequently, the tailored prompts may fail to instruct the model to discern diverse visual concepts from local regions, thereby hindering its capability to generalize across classes that share similar visual attributes. To overcome this limitation, recent works such as ArGue ( Tian et al. 2024 ) and LLaMP ( Zheng et al. 2024 ) utilize large language models (LLMs) to enrich the text prompts with fine-grained class descriptions that enhance transfer performance, as shown in Figure 1 (left). Unfortunately, these approaches necessitate additional processes to sift through the LLMs outputs to ensure their relevance to downstream datasets, thereby introducing significant inference overhead. This work aims to enrich VLM prompt tuning without the reliance on external knowledge experts, as previously described. Our core impetus stems from a widely accepted consensus in the traditional computer field: fine-grained visual concepts at local regions are prominently present in the intermediate layer outputs of the visual networks ( Zeiler and Fergus 2014 ; Kim, Nam, and Ko 2022a ; Selvaraju et al. 2020 ) . As such, one can intuitively exploit the internal knowledge within the VLM, specifically the intermediate outputs of the image encoder, to deliver precise class descriptions and achieve local alignment without the need for external LLMs assistance. To this end, we propose a plug-and-play method, dubbed TextRefiner, to refine the text prompt in VLM tuning without the need for external LLMs’ assistance. As depicted in Fig 1 , TextRefiner incorporates a novel local cache module to encapsulate fine-grained visual concepts derived from local tokens within the image branch. More particularly, amidst VLM tuning, the image branch continuously writes fine-grained semantic information into the cache in a class-wise manner. The fine-grained information retrieved for each category is concatenated with the corresponding class embedding, followed by a feature aggregation module to fuse global and local information to enhance the representation capability for the text branch. To further align the intermediate embeddings between modalities, we furnish TextRefiner with a feature alignment module to transform local tokens into the text embedding space. Owing to such design, the fine-grained local features from the image branch can be seamlessly utilized to deliver class descriptions, thereby effectively refining the vanilla text prompting performance. It is noteworthy that TextRefiner is both scalable and orthogonal to augment the efficacy of prevailing VLM tuning methods, i.e , the local cache module operates independently with existing prompt tuning paradigms. For example, TextRefiner enhances CoOp by 5.30% on 11 benchmarks, surpassing CoCoOp which introduced instance-specific features for text prompts to improve generalization. Equipped with TextRefiner, PromptKD achieves state-of-the-art performance, surpassing LLaMP, which relies on external experts, by 1.06%. Moreover, TextRefiner demonstrates high inference efficiency. CoCoOp achieves 20.45 FPS, LLaMP reaches 1473.46 FPS, while PromptKD w/ TextRefiner achieves 12793.26 FPS. This efficiency is due to its reliance on simple matrix multiplication at the text output, unlike CoCoOp and LLaMP, which require additional modules to extract and combine information with the text input. The contributions of this paper are delineated as follows: • We introduce TextRefiner, which, for the first time, exploits the internal knowledge of the image branch to refine the text prompts of VLMs. • TextRefiner introduces three principal innovations, encompassing local cache, feature aggregation, and feature alignment, which collaboratively enhance the incorporation of fine-grained visual attributes into the text prompts. • Extensive experiments demonstrate the effectiveness