O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models
Ashshak Sharifdeen, Muhammad Akhtar Munir, Sanoojan Baliah, Salman Khan, Muhammad Haris Khan · 2025
Read O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models by Ashshak Sharifdeen, Muhammad Akhtar Munir, Sanoojan Baliah, Salman Khan, Muhammad Haris Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12096.txt.
Introduction
Ashshak Sharifdeen and co-authors fix overconfident image-text models that adapt at test time in this 2025 paper. Prompt tuning lets such models learn from unlabeled images on the fly, boosting accuracy. Yet tuned models often state wrong answers with high confidence, a calibration flaw that undermines trust.
Earlier fixes scattered the text features for each class, hoping confidence would spread honestly. The authors show that approach underperforms, and that forcing the features into a strictly perpendicular arrangement works better. Across datasets and model backbones, their method cut average calibration error well below prior best results. It even beat the untouched model's calibration on fine-grained tasks, a rare double win.
Catalog record
| CREATOR | Ashshak Sharifdeen, Muhammad Akhtar Munir, Sanoojan Baliah, Salman Khan, Muhammad Haris Khan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12096
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models Ashshak Sharifdeen, Muhammad Akhtar Munir, Sanoojan Baliah, Salman Khan, Muhammad Haris Khan Abstract Test-time prompt tuning for vision-language models (VLMs) is getting attention because of their ability to learn with unlabeled data without fine-tuning. Although test-time prompt tuning methods for VLMs can boost accuracy, the resulting models tend to demonstrate poor calibration, which casts doubts on the reliability and trustworthiness of these models. Notably, more attention needs to be devoted to calibrating the test-time prompt tuning in vision-language models. To this end, we propose a new approach, called O-TPT that introduces orthogonality constraints on the textual features corresponding to the learnable prompts for calibrating test-time prompt tuning in VLMs. Towards introducing orthogonality constraints, we make the following contributions. First, we uncover new insights behind the suboptimal calibration performance of existing methods relying on textual feature dispersion. Second, we show that imposing a simple orthogonalization of textual features is a more effective approach towards obtaining textual dispersion. We conduct extensive experiments on various datasets with different backbones and baselines. The results indicate that our method consistently outperforms the prior state of the art in significantly reducing the overall average calibration error. Also, our method surpasses the zero-shot calibration performance on fine-grained classification tasks. Our code is available at https://github.com/ashshaksharifdeen/O-TPT . 1 Introduction In recent years, vision-language models (VLMs), for example, CLIP [ 35 ] , have shown impressive zero-shot inference abilities in various downstream tasks [ 35 , 18 ] . VLMs align web-scale image-text data to learn a shared embedding space, enabling it to classify instances from novel categories in a zero-shot manner via carefully designed prompt templates. So, constructing these manually designed prompts is crucial for effective zero-shot transfer. However, such manually crafted prompts depend on domain-specific heuristics, which may limit their effectiveness [ 37 ] . Prompt tuning aims to learn prompts using downstream training data [ 47 , 46 ] ; however, learned prompts struggle to generalize beyond the task and training data. In addition, prompt-tuning methods require labeled data, which can be costly or even unavailable. Figure 1 : Comparison of calibration performance (ECE) with C-TPT [ 43 ] and Robust-adapt-Penalty-CTPT [ 29 ] . Lower the ECE better the calibration. Recently, test-time prompt tuning [ 37 ] has emerged as a paradigm for refining prompts during inference, enabling VLMs to better adapt to newer tasks without labeled data and retraining. However, while prompt tuning at the test-time has been shown to enhance the accuracy of the VLM, it usually results in poorly calibrated predictions, which means that the confidence of the model does not align well with its accuracy [ 29 , 43 ] . This lack of calibration raises concerns about the trustworthiness of VLMs in applications that demand reliable uncertainty estimates, such as healthcare [ 17 , 40 ] or autonomous systems [ 4 , 44 , 48 ] . Poorly calibrated predictions can lead to overconfidence, which can lead to undesirable consequences in safety-critical applications, where decisions often depend on model outputs. We note that calibration in test-time prompt tuning for VLMs remains largely unexplored. We note that calibration in test-time prompt tuning for VLMs remains largely unexplored with very few attempts. A recent method C-TPT [ 43 ] observes that test-time prompt tuning methods [ 37 ] often sacrifice the model’s calibration performance while boosting its accuracy, leading to unreliable prediction confidence for possibly many test examples. It shows that enhancing text feature dispersion can mitigate the overconfidence issue arising in test-time prompt tuning. However, the method could struggle to ensure the desired dispersion in challenging cases. This can lead to the underutilization of the textual feature space, potentially resulting in textual features lying in close proximity (Fig. 5 ) and eventually causing poor calibration performance (Fig. 1 ). We note that the core methodology of C-TPT [ 43 ] overlooks the critical relationship between the angular separation among textual features and the calibration performance. Our findings reveal a strong correlation: textual features with lower cosine similarity (i.e., greater angular separation) between them lead to an improved calibration, as indicated by a lower Expected Calibration Error (ECE) (Fig. 5 ). Armed with this insight, we propose to impose orthogonalization constraints on the textual features by enforcing the angular distance between them. As such, this allows us to effectively utilize the feature space. Due to improved text feature separation, we observe superior calibration performance (Fig. 1 ). Our contributions are summarized as follows: • We provide new insights underlying the suboptimal performance of an existing top-performing calibration method for the prompt testing-time tuning of VLMs. • We propose a novel approach (named O-TPT ) for calibrating test-time prompt tuning of VLMs by enforcing orthogonality constraints. This is accomplished by introducing orthogonal regularization on the textual features. • We perform an extensive evaluation to validate our approach in various data sets and in different baselines. The results reveal that our O-TPT delivers consistent gains over the state-of-the-art methods in overall average calibration performance with several different baselines. Moreover, our O-TPT provides better calibration performance than the zero-shot CLIP, which reveals improved calibration compared to existing SOTA. 2 Related Work Prompt tuning for vision-language models (VLMs): VLMs like CLIP [ 35 ] and ALIGN [ 18 ] leveraged large datasets of image-text pairs to create shared embedding spaces that align images with their textual descriptions [ 25 ] . This cross-modal alignment enables impressive zero-shot performance across various domains by applying hand-crafted prompt templates. As a result, models can predict without additional training on new tasks. The embeddings are optimized through contrastive learning by matching image-text pairs in a latent space, which enhances the model’s ability to understand the relationship between visual and textual data. Since the design of handcrafted prompts requires domain-specific heuristics, they could be suboptimal for various newer domains. This led to the development of prompt-tuning methods that treat prompts as trainable embeddings. A notable example is CoOP [ 47 ] , which refined prompts in CLIP [ 35 ] using labeled samples, leading to better classification performance. Building upon this, CoCoOP [ 46 ] addressed the limitations of CoOP [ 47 ] by enhancing generalization to out-of-distribution data, effectively improving the model’s ability to new, unseen domains. Recent advancements in prompt tuning, such as Test-Time Prompt Tuning (TPT) [ 37 ] , have introduced algorithms to refine prompts on-the-fly using a single example during inference using entropy minimization. TPT has successfully enhanced model accuracy in zero-shot scenarios. However, it ends up worsening the calibration performance by making overconfident predictions. Calibration of deep neural networks: Calibration evaluates how closely a model’s predicted confidence aligns with its accuracy. It is crucial for high-stakes applications that demand trustworthy uncertainty estimates, such as healthcare and autonomous systems [ 7 ] . Post hoc calibration techniques, such as Temperature Scaling (TS) [ 8 ] , are employed to improve prediction reliability by scaling the model’s logits using a temperature variable. This temperature variable is learned on a hold-out validation set. However, such post-hoc methods often depend on the availability of labeled datasets that closely match the target data distribution [ 21 ] . Such