Zip : An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models
Seonghwan Park, Jaehyeon Jeong, Yongjun Kim, Jaeho Lee, Namhoon Lee · 2025
Read Zip : An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models by Seonghwan Park, Jaehyeon Jeong, Yongjun Kim, Jaeho Lee, Namhoon Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06838.txt.
Introduction
Seonghwan Park and co-authors present ZIP in 2025, a thrifty way to tune prompts for black-box vision-language models. Such models are used through an API, so their insides cannot be trained directly. Past tuning methods need very many queries, which real services limit.
ZIP shrinks the tuning problem by writing prompts in a low-rank form and clipping noisy gradient guesses. That lowers both size and noise, so training runs fast with far fewer queries. Tests on more than a dozen tasks show strong results under tight budgets. Teams that rent model access can adapt behavior without breaking query caps.
Catalog record
| CREATOR | Seonghwan Park, Jaehyeon Jeong, Yongjun Kim, Jaeho Lee, Namhoon Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zip : An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models Seonghwan Park, Jaehyeon Jeong, Yongjun Kim, Jaeho Lee, Namhoon Lee Abstract Recent studies have introduced various approaches for prompt-tuning black-box vision-language models, referred to as black-box prompt-tuning (BBPT). While BBPT has demonstrated considerable potential, it is often found that many existing methods require an excessive number of queries ( i . e ., function evaluations), which poses a significant challenge in real-world scenarios where the number of allowed queries is limited. To tackle this issue, we propose Z eroth-order I ntrinsic-dimensional P rompt-tuning ( Zip ), a novel approach that enables efficient and robust prompt optimization in a purely black-box setting. The key idea of Zip is to reduce the problem dimensionality and the variance of zeroth-order gradient estimates, such that the training is done fast with far less queries. We achieve this by re-parameterizing prompts in low-rank representations and designing intrinsic-dimensional clipping of estimated gradients. We evaluate Zip on 13+ vision-language tasks in standard benchmarks and show that it achieves an average improvement of approximately 6% in few-shot accuracy and 48% in query efficiency compared to the best-performing alternative BBPT methods, establishing a new state of the art. Our ablation analysis further shows that the proposed clipping mechanism is robust and nearly optimal, without the need to manually select the clipping threshold, matching the result of expensive hyperparameter search. 1 Introduction Foundation models pre-trained on a vast amount of data are creating tremendous success across a wide range of applications in various domains ( Ramesh et al., 2021 ; Radford et al., 2021 ; Jia et al., 2021 ; Singh et al., 2022 ; Liu et al., 2023a ; Copet et al., 2024 ) . A notable example is CLIP ( Radford et al., 2021 ) which learns visual concepts from natural language supervision and works zero-shot at inference. In fact, these models are fine-tuned for specific downstream tasks at deployment to create yet more performance refinement in practice ( Liu et al., 2022 ) . It is noteworthy, however, that fine-tuning these models is not only computationally expensive, but also requires full access to model specifications. The complication here is that many high-performing foundation models are provided only as a software-as-a-service ( OpenAI, 2023 ; Google, 2023 ) without model details due to commercial interests and security concerns. To overcome this challenge, recent works have suggested to fine-tune such black-box models via so-called black-box prompt-tuning (BBPT) ( Sun et al., 2022b ; Diao et al., 2023 ; Oh et al., 2023 ; Yu et al., 2023 ) ; i . e ., by parameterizing input prompts and only optimizing them via classic derivative-free optimization methods such as evolutionary strategies ( Hansen & Ostermeier, 2001 ; Hansen et al., 2003 ) and zeroth-order optimization ( Spall, 1992 ; Spall, 1997 ; Ghadimi & Lan, 2013 ) , it enables fine-tuning without direct access to the model details or back-propagation. Nevertheless, it still remains a major challenge to secure query-efficiency in existing BBPT methods. Specifically, we find that many existing approaches require excessive model evaluations ( i . e ., queries), often spanning several tens of thousands times ( Tsai et al., 2020 ; Oh et al., 2023 ) , and moreover, they result in significant performance drop when they are given a limited query budget. This is quite critical in many practical scenarios where large models are provided in the form of prediction APIs, and users can only make use of it with a limited budget. In this work, we propose a new method called Zip : Z eroth-order I ntrinsic-dimensional P rompt-tuning to tackle this challenge. The key idea is to reduce the dimensionality of the problem (hence the term “intrinsic”) and the variance of zeroth-order gradients, such that the training is done fast with far less queries, and subsequently, improves generalization. In essence, we achieve this by re-parameterizing prompts in low-rank representations in effective forms and designing intrinsic-dimensional clipping of zeroth-order gradients ( Section 4 ). Fundamentally, we are inspired by a line of previous works that hint at the dimension dependency of zeorth-order methods ( Spall, 1992 ; Ghadimi & Lan, 2013 ; Duchi et al., 2015 ) and noise in stochastic methods ( Bottou et al., 2018 ) causing optimization difficulty when it comes to large models. Indeed, we find this pertinent to BBPT in our empirical analysis, in which we show that a naive zeroth-order method suffers from increased dimensionality unlike the first-order counterpart ( Section 3 ). Our extensive experimental results show that Zip is extremely efficient and robust, setting a new state of the art. To be specific, we evaluate Zip on 13+ datasets for various vision-language tasks in standard benchmarks including few-shot learning, base-to-new generalization, cross-dataset transfer, and out-of-distribution generalization tasks, and across all, we find that Zip consistently outperforms existing BBPT methods by substantial margins in terms of prediction accuracy, while demanding far less number of queries ( Section 5 ). We provide a summary of how Zip performs in Figure 1 . (a) Query efficiency (b) Training progress (c) Generalization performance Figure 1: Zip performance summary: (a) number of queries required to reach a given target accuracy, (b) training accuracy vs. number of queries, and (c) test set accuracies, measured on 13 standard vision-language tasks; the first two are arithmetic mean. Zip shows its outstanding performance compared to other black-box prompt-tuning methods ( e . g ., BAR ( Tsai et al., 2020 ) , BlackVIP ( Oh et al., 2023 ) , BPTVLM ( Yu et al., 2023 ) ; the details are provided in Section C.5 ) in terms of both training and generalization performances. 2 Background Prompt-tuning is an emerging paradigm to update large pre-trained models before utilizing them for various downstream tasks ( Lester et al., 2021 ; Liu et al., 2023b ) . It works by prepending learnable context tokens to the input prompts embedding and training them on some data for a target task. Specifically, we can formulate it as an optimization problem as follows: \min_{\theta}f(\theta,\omega;\mathcal{D}) (1) where \theta\in\mathbb{R}^{d} refers to the learnable parameters where d=p\times m denotes the problem dimensionality with p and m being the word embedding dimensions and number of context tokens, respectively, and \omega refers to the pre-trained model; also, f refers to the loss function, which is the cross-entropy for vision-language tasks in our case, and \mathcal{D} is a given dataset. However, prompt-tuning is not directly applicable to black-box models since back-propagation is not allowed, and thus, the optimization must be done derivative-free. To this end, many approaches have been developed to address this issue for namely derivative-free optimization ( Larson et al., 2019 ) . In particular, a series of evolution strategies such as covariance matrix adaptation ( Hansen, 2016 ) has been used for black-box prompt tuning or BBPT ( Sun et al., 2022b ; Sun et al., 2022a ; Yu et al., 2023 ) . An alternative approach to prompt-tuning in a black-box situation is by zeroth-order optimization which leverages approximate gradients estimated from only function evaluations. In particular, one of the foundational techniques is the simultaneous perturbation stochastic approximation (SPSA) ( Spall, 1992