FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation Journal: Expert Systems with Applications

Zherui Zhang, Jiaxin Wu, Changwei Wang, Rongtao Xu, Longzhao Huang, Wenhao Xu, Wenbo Xu, Li Guo, Shibiao Xu · 2025

Read FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation Journal: Expert Systems with Applications by Zherui Zhang, Jiaxin Wu, Changwei Wang, Rongtao Xu, Longzhao Huang, Wenhao Xu, Wenbo Xu, Li Guo, Shibiao Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18053.txt.

Introduction

Zherui Zhang and a large team speed up prompt learning for vision-text models by one shared teaching set. Teaching by a big guide model lifts spread to new classes, yet live guide calls at each stage eat the speed that prompt learning is for. Their 2025 method, FDBPL, shares the soft marks of the guide across stages and speeds the data path. A region-aware form then learns from cut regions of images.

Twin prompt fields, one for true kinship and one for weak kinship, learn by a split rule, so the small model takes true sense and spurns the weak. On 11 datasets the method tops base-to-new, cross-set, and toughness tests. Training ran 2.2-fold faster, with lean size kept.

Catalog record

CREATORZherui Zhang, Jiaxin Wu, Changwei Wang, Rongtao Xu, Longzhao Huang, Wenhao Xu, Wenbo Xu, Li Guo, Shibiao Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18053

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation Journal: Expert Systems with Applications Zherui Zhang, Jiaxin Wu, Changwei Wang, Rongtao Xu, Longzhao Huang, Wenhao Xu, Wenbo Xu, Li Guo, Shibiao Xu Abstract Prompt learning as a parameter-efficient method that has been widely adopted to adapt Vision-Language Models (VLMs) to downstream tasks. While hard-prompt design requires domain expertise and iterative optimization, soft-prompt methods rely heavily on task-specific hard labels, limiting their generalization to unseen categories. Recent popular distillation-based prompt learning methods improve generalization by exploiting larger teacher VLMs and unsupervised knowledge transfer, yet their repetitive teacher model online inference sacrifices the inherent training efficiency advantage of prompt learning. In this paper, we propose F aster D istillation- B ased P rompt L earning ( FDBPL ), which addresses these issues by sharing soft supervision contexts across multiple training stages and implementing accelerated I/O. Furthermore, FDBPL introduces a region-aware prompt learning paradigm with dual positive-negative prompt spaces to fully exploit randomly cropped regions that containing multi-level information. We propose a positive-negative space mutual learning mechanism based on similarity-difference learning, enabling student CLIP models to recognize correct semantics while learning to reject weakly related concepts, thereby improving zero-shot performance. Unlike existing distillation-based prompt learning methods that sacrifice parameter efficiency for generalization, FDBPL maintains dual advantages of parameter efficiency and strong downstream generalization. Comprehensive evaluations across 11 datasets demonstrate superior performance in base-to-new generalization, cross-dataset transfer, and robustness tests, achieving 2.2 × 2.2\times faster training speed. 1 Introduction With the rapid development of deep learning technology, traditional models that rely only on visual features for precise recognition are no longer sufficient to meet the diverse task requirements found in open-world scenarios. Therefore, recent large-scale pre-trained Vision-Language Models (VLMs), such as CLIP [ 1 ] , DALL-E [ 2 ] , and Stable Diffusion [ 3 ] , allow them to flexibly handle various downstream tasks [ 4 ] , including image classification [ 5 , 6 , 7 ] , object detection [ 8 , 9 , 10 ] , semantic segmentation [ 11 , 12 , 13 , 14 ] , Embodied Artificial Intelligence [ 15 , 16 , 17 , 18 , 19 ] and visual question answering [ 20 , 21 ] . Specifically, CLIP employs a dual-tower architecture, comprising an image encoder and a text encoder, which combines natural language processing with computer vision and has completed extensive self-supervised pre-training on the vast amount of image-text pairs. Figure 1 : Training Efficiency Advantage. The results are measured in minutes, and our method demonstrates significant improvements across all 11 datasets, as well as in average training time. Figure 2 : From PL to FDBPL (Ours) . Prompt learning (PL) adapts CLIP to downstream tasks via learnable parameters. We compare three methods: (a) Native PL employs dataset hard labels, offering fast training but suffering from overfitting on seen classes, which degrades zero-shot performance on unseen classes. (b) Distillation-Based PL uses a teacher CLIP network to transfer generalization knowledge without specific labels, often on unlabeled data. However, the online inference required for soft label generation reduces its training efficiency. (c) Our proposed Faster Distillation-Based PL (FDBPL) achieves both high training efficiency and strong zero-shot generalization by pre-storing shareable soft labels across epochs and employing fast I/O. VLMs like CLIP demonstrate remarkable zero-shot abilities, but require adaptation for optimal performance on downstream tasks. Among various fine-tuning explorations, prompt learning has emerged as a parameter-efficient technique that preserves CLIP’s frozen dual encoder architecture and adjusts only the prompt behavior, generally falling into two categories: hard-prompt learning and soft-prompt learning. Standard hard-prompt learning employs fixed templates (e.g., “a photo of a [class name]" ) to guide the text encoder in generating class-specific text features, where visual-textual similarity determines final predictions. However, hard-prompt learning suffers from critical limitations: designing optimal hard-prompts requires extensive expert knowledge or trial-and-error adjustments, making it an art rather than a science [ 22 ] . In contrast, learnable soft-prompt learning addresses this challenge by exploiting few-shot training data to optimize class-wise context vectors through contrastive alignment between visual and textual features in a shared embedding space, demonstrating superior performance over manual template engineering. ProGrad [ 23 ] and KgCoOp [ 24 ] reveal critical limitations in both paradigms: soft-prompts achieve strong in-domain performance in base classes, tend to overfit training data and show a weak generalization to unseen classes [ 25 , 26 ] , and hard-prompts maintain better zero-shot performance through their explicit semantic grounding. LAMM [ 27 ] introduces an innovative label alignment that dynamically adjusts the category embeddings of downstream datasets through end-to-end training. GIST [ 28 ] addresses the behavioral gap between trainable parameters and downstream task knowledge. The above observation raises a critical challenge: how can we effectively exploit the advantages of both hard and soft prompt learning? Consequently, another branch of prompt learning explores the extraction of knowledge from more powerful VLMs as teachers, thereby reducing the need for labeled data for target downstream tasks. Figure 2 (b) demonstrates how prompt learning and knowledge distillation are combined to improve the zero-shot adaptation performance of the student CLIP in downstream tasks. In this paper, we follow the design paradigm of Distillation-Based Prompt Learning, but we emphasize that the inherent efficiency advantages of native prompt learning are impaired by the introduction of knowledge distillation. As depicted in Figure 2 (b) , knowledge distillation, acting as an additional supervisory objective for the student CLIP, introduces a non-negligible efficiency degradation (i.e., longer training time per epoch), due to obtaining such supervisory signals inevitably necessitates online inference from the larger teacher VLMs. In other words, a considerable portion of computational resources is dedicated to feeding training data into the large teacher network to generate supervisory information instead of updating the student CLIP’s prompts, and this supervisory information is repeatedly computed across different training epochs. Therefore, the central problem addressed in this paper is how to leverage knowledge distillation to improve the performance of soft-prompt learning on zero-shot downstream tasks while preserving the native training efficiency advantages. We propose the F aster D istillation- B ased P rompt L earning ( F DBPL ) framework, whose core idea adopts a space-for-time strategy to fast I/O share computations of teacher supervision signals across different training iterations, as shown in Figure 2 (c) . The proposed FDBPL comprises two key components: a Region-Aware Dual-Prompt (RADP) learning strategy and a Prompt-Cascaded Difference (PCD) module. Specifically, we randomly crop regions of interest (ROI) from the given image, store the absolute spatial position information of the cropped ROIs, the type of image transformation, and the corresponding teacher supervision information. However, due to the randomness of cropping, low-information regions inevitably emerge, which may lack the necessary semantic content. Unlike the rough discard operation [ 29 , 30 ] , in the proposed RADP learning, in addition to learning positive semantics through the positive prompt, we introduce learnable negative prompts that establish a greater similarity to cropped regions without explicit semantic information, teaching the student CLIP to say "No". On the other hand, the PCD learning analyze first-order and second-order difference similarities between image and text features, capturing potential 1) intra-class differences between the presence and absence

Read the complete text