A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models

Shihab Aaqil Ahamed, Udaya S.K.P. Miriya Thanthrige, Ranga Rodrigo, Muhammad Haris Khan · 2025

Read A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models by Shihab Aaqil Ahamed, Udaya S.K.P. Miriya Thanthrige, Ranga Rodrigo, Muhammad Haris Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26441.txt.

Introduction

Shihab Aaqil Ahamed, with three co-authors, improves how vision-language models adjust to new tasks at test time. These models link images and text, and test-time prompt tuning tweaks their prompts without labelled data. Weak spread among text features can make their confidence poorly calibrated, meaning stated odds do not match real accuracy.

Ahamed and his co-authors propose A-TPT, which pushes text features apart in angle so they spread evenly on a sphere. Their trick maximises the smallest angle between any pair of class features. Across backbones and data sets, A-TPT cut calibration error while holding accuracy near prior methods. It did especially well under natural shifts in the data and carried over to medical data sets. The paper backs the method with theory as well as tests.

Catalog record

CREATORShihab Aaqil Ahamed, Udaya S.K.P. Miriya Thanthrige, Ranga Rodrigo, Muhammad Haris Khan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models Shihab Aaqil Ahamed, Udaya S.K.P. Miriya Thanthrige, Ranga Rodrigo, Muhammad Haris Khan Abstract Test-time prompt tuning (TPT) has emerged as a promising technique for adapting large vision-language models (VLMs) to unseen tasks without relying on labeled data. However, the lack of dispersion between textual features can hurt calibration performance, which raises concerns about VLMs’ reliability, trustworthiness, and safety. Current TPT approaches primarily focus on improving prompt calibration by either maximizing average textual feature dispersion or enforcing orthogonality constraints to encourage angular separation. However, these methods may not always have optimal angular separation between class-wise textual features, which implies overlooking the critical role of angular diversity. To address this, we propose A-TPT , a novel TPT framework that introduces angular diversity to encourage uniformity in the distribution of normalized textual features induced by corresponding learnable prompts. This uniformity is achieved by maximizing the minimum pairwise angular distance between features on the unit hypersphere. We show that our approach consistently surpasses state-of-the-art TPT methods in reducing the aggregate average calibration error while maintaining comparable accuracy through extensive experiments with various backbones on different datasets. Notably, our approach exhibits superior zero-shot calibration performance on natural distribution shifts and generalizes well to medical datasets. We provide extensive analyses, including theoretical aspects, to establish the grounding of A-TPT. These results highlight the potency of promoting angular diversity to achieve well-dispersed textual features, significantly improving VLM calibration during test-time adaptation. Our code will be made publicly available. 1 Introduction Foundational large-scale vision-language models (VLMs), such as CLIP ( Radford et al. 2021 ) , ALIGN ( Jia et al. 2021 ) , and FILIP ( Yao et al. 2021 ) , have demonstrated remarkable zero-shot inference capabilities in a wide range of downstream tasks ( Jia et al. 2021 ; Radford et al. 2021 ) . These models are pre-trained with contrastive learning on massive web-scale data — e.g., 400 million image-text caption pairs — to align visual and textual modalities within a shared multimodal latent space. This alignment allows VLMs to classify instances from novel visual categories in a zero-shot setting realized by carefully constructed textual prompts — hand–crafted class-conditioned templates (e.g., “a photo of a [class]”) — crucial for effective zero-shot transfer. However, manually designing such prompts often requires domain-specific heuristics and may not be optimal across diverse tasks ( Shu et al. 2022 ) Figure 1: Comparison of calibration performance (ECE) with C-TPT ( Yoon et al. 2024 ) , and O-TPT ( Sharifdeen et al. 2025 ) on fine-grained classification datasets with CLIP ViT-B/16 backbone. Ours (lower ECE) shows improved prompt calibration. Figure 2: Comparison of numerical optimization (A-TPT (Ours)) with angular optimization (O-TPT Sharifdeen et al. 2025 ) and ATFD optimization (C-TPT Yoon et al. 2024 ). To address these limitations, recent works have explored prompt tuning that learns prompts from training data specific to downstream tasks ( Zhou et al. 2022a ; Zhou et al. 2022b ) . However, such approaches often rely on annotated data, which can be expensive and scarce for zero-shot scenarios ( Socher et al. 2013 ) . To address this challenge, test-time prompt tuning (TPT) ( Shu et al. 2022 ) has garnered significant attention focused on prompt tuning. TPT optimizes learnable prompt vectors through gradient descent that adaptively refines them during inference with unlabelled test image samples to adapt VLMs to novel tasks. Although TPT can boost the accuracy of VLMs, it often suffers from poor calibration, where the model’s predicted confidence does not reliably reflect the true accuracy ( Guo et al. 2017 ; Murugesan et al. 2024 ; Yoon et al. 2024 ) . Such miscalibration can result in overconfident predictions, raising concerns about the reliability and trustworthiness of VLMs, particularly for real-world safety-critical applications that require reliable uncertainty estimates, including medical diagnostics ( Ji et al. 2021 ; Wang et al. 2022 ; Zhang et al. 2023 ; Chen et al. 2023 ; Liu et al. 2023 ) and autonomous driving ( Dorbala et al. 2022 ; Gadre et al. 2022 ; Khandelwal et al. 2022 ; Bucker et al. 2023 ; Cui et al. 2024 ; You et al. 2024 ; Zhou et al. 2024 ) . To date, calibration in test-time prompt tuning of VLMs is less explored, with limited efforts to address it. To do better prompt calibration, prior works, such as C-TPT ( Yoon et al. 2024 ) and O-TPT ( Sharifdeen et al. 2025 ) have explored methods to encourage dispersion between pairwise textual features, which can be categorized into the following two types: The first type, known as Average Textual Feature Dispersion (ATFD), spreads textual features away from their centroid. However, this can still result in textual features lying closely together (Fig. 2 ) and cause poor calibration performance (Fig. 2 ). The second type enforces orthogonality constraints to encourage angular separation, which exploits an auxiliary orthogonal regularization term in the loss function to encourage pairwise textual features as orthogonal as possible. However, we observe that it tends to group textual features closer, particularly when the number of classes N is greater than the embedding dimension |D| (e.g., \left(N>|D|\right) , where CLIP’s ViT-B/16 512-d ( Radford et al. 2021 ; Liang et al. 2022 ) vs. 1000 classes in ImageNet-1k, V2, K) does not guarantee uniformity of angular separation (Fig. 2 , 4 a). When the number of classes is less than the embedding dimension ( \left(N<|D|\right) , e.g., classes: 10 in EuroSAT, 37 in OxfordPets, and 47 in DTD), it fails to fully utilize the hyperspherical space of feature points effectively across the hypersphere (Fig. 2 , 4 b). This eventually leads to poor calibration (Fig. 2 ). Although these methods increase feature dispersion to some extent, they often overlook the importance of angular diversity. Without sufficient angular separation, prompts may become highly correlated, which limits the model’s ability to generate well-calibrated predictions. Prior work ( Wang & Isola 2020 ) has shown that uniformity (uniformly distributed feature points on the unit hypersphere) preserves maximal information, closely associated with strong zero-shot CLIP performance. The uniformity problem is well-studied in Tammes problem ( Tammes 1930 ) (best-packing), that is to find the optimal arrangement of a given number of feature points on the surface of a unit hypersphere such that the minimum distance between any two points is maximized. Inspired by this insight, we propose A-TPT , a numerical optimization approach that introduces a simple yet effective angular diversity into the test-time prompt tuning framework. Our method maximizes the minimum pairwise angular distance between normalized textual features on the unit hypersphere to promote uniform and diverse prompt distribution, fully utilizing the hyperspherical space (Fig. 2 ). By penalizing closely aligned prompt directions, the proposed A-TPT promotes the greatest possible angular distance between them (Fig. 3(b) ), thus achieving better prompt calibration performance (Fig. 2 ). Notably, by maximizing the minimum angular distance between prompt vectors, A-TPT naturally solves the number of classes exceeding the embedding dimension problem, e.g., 1000 classes in a 512-d space, making all prompt vectors pairwise orthogonal impossible. In such cases, hypersphere can still achieve a good class separation in

Read the complete text