ANPrompt: Anti-noise Prompt Tuning for Vision-Language Models
Yansheng Gao, Yufei Zheng, Jinghan Qu, Zixi Zhu, Yukuan Zhang, Shengsheng Wang · 2025
Read ANPrompt: Anti-noise Prompt Tuning for Vision-Language Models by Yansheng Gao, Yufei Zheng, Jinghan Qu, Zixi Zhu, Yukuan Zhang, Shengsheng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04677.txt.
Introduction
Yansheng Gao and co-authors present ANPrompt, a way to tune the prompts of vision-language models so they stay steady when wording shifts slightly, in a 2025 paper. Prompt tuning adjusts a model's guiding text rather than the model itself. Past work tried to scrub all noise out of those prompts, which the team says backfires, making the model brittle on new categories.
Instead, ANPrompt feeds the model a controlled dose of weak noise. It clusters softly scrambled features into noise prompts and mixes them with the learned tokens on both the text and image sides. A prototype step steadies the image side, and a loss term keeps answers on clean and scrambled inputs in agreement. Across eleven test sets, the method beat earlier prompt tuning on both toughness and reach to new classes.
Catalog record
| CREATOR | Yansheng Gao, Yufei Zheng, Jinghan Qu, Zixi Zhu, Yukuan Zhang, Shengsheng Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04677
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ANPrompt: Anti-noise Prompt Tuning for Vision-Language Models Yansheng Gao, Yufei Zheng, Jinghan Qu, Zixi Zhu, Yukuan Zhang, Shengsheng Wang Abstract Prompt tuning has emerged as an efficient and effective technique for adapting vision-language models (VLMs) with low computational overhead. However, existing methods often overlook the vulnerability of prompt-tuned VLMs to weak semantic perturbations—such as subtle image or text noise—that degrade their generalization to unseen classes. To address this limitation, we propose ANPrompt, a novel prompt tuning framework designed to enhance robustness under such perturbations. ANPrompt first constructs weak noise text features by fusing original and noise-perturbed text embeddings, which are then clustered to form noise prompts. These noise prompts are integrated with learnable prompt tokens to generate anti-noise prompts, which are injected into the deeper layers of both image and text encoders. To further capture the noise-aware visual semantics, ANPrompt computes the Noise-Resistant Visual Prompt Prototype (NRVPP) by averaging the output prompt tokens from the vision encoder. Finally, ANPrompt introduces alignment, robustness, and anti-noise objectives by computing a Weak semantic noise Alignment Loss (WALoss) alongside the standard cross-entropy and sim loss. Experiments across 11 benchmarks demonstrate that ANPrompt consistently outperforms existing prompt tuning approaches, achieving superior robustness to semantic noise and improved generalization to novel categories. Introduction Vision-language models (VLMs) like CLIP have demonstrated impressive performance on open-vocabulary recognition tasks ( Hua et al. 2025 ; Weng et al. 2023 ; Xu et al. 2024 ) , but adapting such large models to downstream tasks via full fine-tuning can be computationally prohibitive. To address this, prompt tuning ( Zhu et al. 2024 ; Lafon et al. 2024 ; Shi et al. 2024 ) has emerged as a lightweight alternative by optimizing only a small set of learnable tokens, achieving state-of-the-art results across various domains. Despite its efficiency, early prompt tuning methods such as CoOp ( Zhou et al. 2022b ) often overfit to seen classes, resulting in limited generalization to novel categories and vulnerability to subtle semantic perturbations. To enhance transferability, recent approaches have integrated inductive biases into the prompting process. For instance, CoCoOp conditions prompts on image features to mitigate overfitting; KgCoOp constrains semantic drift by leveraging frozen text embeddings; and MaPLe adopts multimodal prompts to achieve more robust alignment. More advanced techniques such as PromptSRC and CoPrompt regularize prompt learning using frozen zero-shot logits, while MMRL and TAC construct shared or clustered semantic spaces. Although designed to reduce overfitting, consistency constraints based on frozen features may introduce noise information such as misleading text or visual clutter caused by domain gaps, ultimately weakening the robustness of Vision-Language Models. Figure 1: Weak semantic perturbations in image and text: CAMs may highlight irrelevant regions (e.g., background or co-occurring objects), while subtle textual cues (e.g., “dark stripes”) may mislead class semantics (e.g., shift from “cat” to “tiger”). Figure 2: Different with existing methods use frozen logits as constraint, risking overfitting under weak semantic perturbations. ANPrompt introduces weak semantic perturbations to generate soft logits and anti-prompt to improves generalization. As shown in Figure 1 , we define subtle noise in images and texts as weak semantic perturbations, which appear as either misleading text (e.g., “a cat resembling a small tiger”) or visual noise (e.g., background clutter or co-occurring objects). While high-quality image-text pairs exhibit clear object focus and accurate descriptions, existing prompt tuning methods often struggle to handle such subtle shifts, leading to significant performance drops when adapting to novel categories. This highlights the urgent need for VLMs capable of learning robust, noise-resilient cross-modal representations to maintain alignment consistency and improve generalization. To address the issue of weak semantic perturbations, we propose ANPrompt — a prompt tuning framework specifically designed to enhance robustness under such perturbations. As illustrated in Figure 2 , existing methods rely on frozen feature constraints and thus tend to suffer from overfitting when exposed to weak semantic perturbations. In contrast, our ANPrompt framework intentionally incorporates weak semantic perturbation information into prompt tokens and logits, thereby boosting the robustness of both the learned prompts and class distributions in scenarios involving weak semantic perturbations. Specifically, ANPrompt randomly samples two intra-class descriptions from a language cache: a main text that conveys core semantics, and a noise text that introduces minor linguistic variations. Their fusion generates a Weak Noise Frozen Text Feature , which incorporates weak semantic perturbation information while preserving class identity. These features are clustered into noise prompts and injected as biases into learnable tokens, forming Anti-noise Prompts . These prompts are then inserted into the higher layers of both text and vision encoders to introduce weak semantic perturbation information at the prompt representation level. After encoding, ANPrompt extracts image Feature, text Feature, and the Noise-Resistant Visual Prompt Prototype (NRVPP) . The latter is obtained by mean-pooling the visual prompt tokens from the final layer of the visual encoder. Since Anti-noise Prompts are injected into deeper encoder layers, the resulting NRVPP inherit weak semantic perturbations. To stabilize training under both weak semantic perturbations, we introduce a KL divergence-based Weak Alignment Loss (WALoss) between Weak Noise Logits and Logits set, which enforces alignment consistency under controlled weak semantic perturbations. Additionally, we employ a CE and sim loss to enable the model to learn task-relevant knowledge under weak semantic perturbations. By combining these losses, ANPrompt can improve the robustness of VLMs under weak semantic perturbations. Extensive experiments conducted on 11 benchmarks and base-to-new splits show that ANPrompt significantly outperforms existing prompt tuning methods, achieving state-of-the-art performance in terms of robustness and generalization. Our main contributions are summarized as follows: • We propose ANPrompt , a prompt tuning framework that improve the robustness of VLMs by injecting weak semantic perturbations into both prompt tokens and logits pathways. • We introduce a Weak semantic noise Alignment Loss (WALoss) to enforce alignment consistency under controlled weak semantic perturbations. • We develop a weak semantic perturbations prompt injection mechanism that clusters weak noise frozen text features into noise prompts and injects them into learnable tokens to get anti-noise prompt, which is used to inject weak semantic perturbations at the prompt representation level. • ANPrompt achieves state-of-the-art performance on 11 benchmark datasets under base-to-new generalization settings. Related Work Vision-Language Models Pretrained vision-language models (VLMs) such as CLIP ( Radford et al. 2021 ) and ALIGN ( Jia et al. 2021 ) achieve strong open-vocabulary and cross-modal capabilities through contrastive learning on large-scale image-text pairs. Building on this foundation, models like BLIP ( Li et al. 2022 ) , FILIP ( Yao et al. 2021 ) , Florence ( Yuan et al. 2021 ) , and LiT ( Zhai et al. 2022 ) enhance fine-grained alignment and data efficiency via retrieval augmentation ( Li et al. 2024a ) , self-supervised learning, and stronger backbones. NLIP ( Huang et al. 2023 ) further improves robustness by introducing noise-harmonization based on alignment uncertainty and noise-completion via concept-guided caption generation. These advances enable effective zero-shot and few-shot transfer for tasks such as retrieval, classification, and grounding. However, adapting such large models to downstream tasks ( Li et al. 2025a ; Li et al. 2023 ) remains costly due to domain shifts and the high overhead of full fine-tuning. To address this, parameter-efficient tuning methods—such as adapter-based approaches ( Gao et al. 2024