Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo \corresponding · 2025

Read Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment by Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo \corresponding (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19060.txt.

Introduction

Song-Lin Lv and co-authors proposed a training-free way to fix the confidence of tuned vision-language models in a 2025 paper. Tuning a model like CLIP can leave its stated confidence out of line with its accuracy, above all on classes it never saw. Past fixes leaned on text features of the training classes, which limits their reach.

Their method compares the raw scores of the tuned model with those of the base model for each sample. That gap, the logit shift, is turned into a scaling factor that adjusts confidence while keeping the predicted class. Across eleven data sets and five tuning methods, the approach improved confidence marks on seen and unseen classes alike. A further test tied the shift signal to error size, backing the design.

Catalog record

CREATORSong-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo \corresponding
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19060

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo \corresponding Abstract Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes. Existing VLM-specific calibration methods mainly rely on textual features of train classes, limiting their applicability across class and distribution shifts. We propose Shift-Aware Calibration (SAC) , a training-free, sample-wise method that uses the discrepancy between the output logits of the original and fine-tuned CLIP as a calibration signal. We explicitly call this quantity logit shift , since it is measured in prediction space rather than between hidden representations. SAC maps this logit shift to a positive scaling factor that adjusts confidence while preserving the predicted class. Across 11 datasets and five fine-tuning methods, SAC improves calibration on train and unseen classes and under cross-dataset and domain-generalization evaluation. Mutual Information (MI) analysis shows a statistical dependency between the logit-shift signal and ECE across methods and datasets, providing empirical evidence that the signal used by SAC is informative for calibration and supporting the effectiveness of its shift-aware design. 1 School of Intelligence Science and Technology, Nanjing University, China 2 National Key Laboratory for Novel Software Technology, Nanjing University, China [email protected] Introduction Vision-language models, such as CLIP ( Radford et al. 2021 ) , pre-trained on vast web-scale text-image datasets, have demonstrated impressive zero-shot capabilities and image-text alignment in downstream image classification tasks ( Deng et al. 2009 ; Helber et al. 2019 ) . Concurrently, various prompt learning methods for VLMs have been proposed to further enhance the performance of VLMs on specific tasks by leveraging a small amount of labeled data ( Khattak et al. 2023a ; Khattak et al. 2023b ; Zhou et al. 2022b ) . Given CLIP’s strong zero-shot adaptability, the open-vocabulary setting has become a standard for evaluating the performance of fine-tuned VLMs, where prompts are trained on a subset of classes and evaluated on both train and unseen classes ( Lee et al. 2023 ; Tan et al. 2024 ) . Unfortunately, fine-tuned VLMs regularly overfit to train classes, forgetting the well-calibrated predictions and image-text alignment achieved during pre-training ( Zhou et al. 2022a ; Zhou et al. 2022b ) . For unseen classes, they often produce semantically unbalanced representations, leading to image-text misalignment and a significant discrepancy between confidence scores and actual accuracy ( Guo et al. 2017a ; Minderer et al. 2021 ) . Existing calibration methods ( Joy et al. 2023 ; Oh et al. 2023 ; Zadrozny and Elkan 2001 ) typically fit or analyze statistics from the training distribution, which limits their use on unseen classes. Distance-Aware Calibration (DAC) ( Wang et al. 2024 ) , a VLM-specific method, derives class-wise weights from textual representations and is designed primarily for unseen classes within the same dataset; it does not directly model sample-level visual shifts in cross-dataset or domain-generalization evaluation. Prior work ( Minderer et al. 2021 ; Tu et al. 2023 ; Khattak et al. 2023b ) suggests that the original CLIP benefits from well-aligned image–text representations and often exhibits favorable calibration. Motivated by this observation, we analyze fine-tuning dynamics and find that changes in calibration are associated with a mismatch between the shift in model outputs and the corresponding change in accuracy. Because our measurable quantity is computed from logits rather than hidden features, we refer to it as logit shift ; it serves as an observable proxy for the prediction-space effects of fine-tuning. Based on this analysis, we develop Shift-Aware Calibration (SAC) . SAC uses the discrepancy between the logits of the original CLIP (the “calibration anchor”) and the fine-tuned CLIP to compute a sample-wise scaling coefficient. The coefficient is positive, so rescaling changes confidence without changing the predicted class. Due to its design tailored for CLIP with strong multimodal alignment, SAC can be applied to any set of classes and any evaluation settings, such as open-vocabulary, cross-dataset, and domain-generalization across 11 datasets. Using default parameters, SAC consistently delivers calibration improvements across all five fine-tuning methods and achieves the best performance in confidence calibration, outperforming the current best calibration method MIR ( Roelofs et al. 2022 ) and SOTA VLM-specific calibration method DAC ( Wang et al. 2024 ) . Notably, SAC demonstrates robust performance across various potential variations, including training sample shots, model backbones, datasets, fine-tuning methods, and evaluation settings, highlighting its insensitivity to parameters. In summary, the main contributions of this paper include: • We identify that the key driving factor of confidence imbalance of fine-tuned CLIP is the mismatch between the degree of logit shift and the corresponding changes in accuracy. This finding provides a reliable reference for future research aimed at addressing confidence calibration issues in fine-tuned models. • Based on the analysis, we propose a novel method, Shift-Aware Calibration (SAC). As a post-hoc and fine-grained calibration approach, SAC achieves sample-level confidence calibration without compromising the model’s output accuracy. • Benefiting from the design of using the original CLIP as a “calibration anchor”, SAC overcomes the limitations of existing methods and extends the experimental scope to all VLM evaluation settings, including open-vocabulary, cross-dataset, and domain-generalization. • The experimental results on 11 datasets, testing the calibration of 5 fine-tuning methods, show that SAC outperforms existing calibration methods. SAC demonstrates strong robustness and parameter insensitivity. Related Works Prompt Learning in Vision Language Models. Due to the large parameter size of VLMs and the limited availability of training data for downstream tasks, it is impractical to fine-tune all parameters of the VLMs to adapt them to these tasks. To improve the generalization of the learnable language prompt ( Zhou et al. 2022b ; Zang et al. 2024 ; Mistretta et al. 2024 ) , CoCoOp ( Zhou et al. 2022a ) generates a vision-conditional prompt by fusing the image feature and the learnable language prompts. KgCoOp ( Yao et al. 2023 ) and ProGrad ( Zhu et al. 2023 ) are other prompt-based methods for VLMs. MaPLe ( Khattak et al. 2023a ) and PromptSRC ( Khattak et al. 2023b ) conduct the visual-textual prompt learning by jointly conducting the prompt learning on the vision and text encoders. To address higher confidence bias for unseen classes caused by fine-tuning, the proposed method leverages the original CLIP to correct the confidence of fine-tuned CLIP. Confidence Calibration. Confidence calibration aims to align the confidence scores predicted by models with their actual performance. A common strategy for achieving this is to apply calibration techniques after model training. These techniques can be broadly divided into two categories: scaling-based methods ( Guo et al. 2017a ; Xiong et al. 2023 ; Guo et al. 2017b ) and bin-based methods ( Roelofs et al. 2022 ; Zadrozny and Elkan 2001 ; Zadrozny and Elkan 2002 ) . Distance-Aware Calibration ( Wang et al. 2024 ) estimates class-wise scaling weights for unseen classes from textual representations, but its original formulation is limited when the target class vocabulary or visual distribution changes substantially. As a newly proposed method, SAC enables effective sample-level calibration across common experimental scenarios in VLM research, with a detailed comparison to existing methods provided in Table 1 .

Read the complete text