PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models

Mayank Nautiyal, Stela Arranz Gheorghe, Kristiana Stefa, Li Ju, Ida-Maria Sintorn, Prashant Singh · 2025

Read PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models by Mayank Nautiyal, Stela Arranz Gheorghe, Kristiana Stefa, Li Ju, Ida-Maria Sintorn, Prashant Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11360.txt.

Introduction

Mayank Nautiyal and five co-authors improve how language can steer image classification. Models that join vision and language can point attention at the parts of an image a phrase names. But fixed embeddings treat that pointing as certain, when the match between words and pixels is often ambiguous. Their 2025 framework, PARIC, makes the reference attention probabilistic instead, spreading belief over plausible regions. A regularization step then guides the classifier's own attention toward that spread.

The change should help where one phrase fits many layouts, or where labels are noisy. It keeps the interpretability that made language guidance attractive. The authors test the idea on standard classification tasks. Their work adds a dose of honest uncertainty to a popular design pattern.

Catalog record

CREATORMayank Nautiyal, Stela Arranz Gheorghe, Kristiana Stefa, Li Ju, Ida-Maria Sintorn, Prashant Singh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11360

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models Mayank Nautiyal, Stela Arranz Gheorghe, Kristiana Stefa, Li Ju, Ida-Maria Sintorn, Prashant Singh Abstract Language-guided attention frameworks have significantly enhanced both interpretability and performance in image classification; however, the reliance on deterministic embeddings from pre-trained vision-language foundation models to generate reference attention maps frequently overlooks the intrinsic multivaluedness and ill-posed characteristics of cross-modal mappings. To address these limitations, we introduce PARIC, a probabilistic framework for guiding visual attention via language specifications. Our approach enables pre-trained vision-language models to generate probabilistic reference attention maps, which align textual and visual modalities more effectively while incorporating uncertainty estimates, as compared to their deterministic counterparts. Experiments on benchmark test problems demonstrate that PARIC enhances prediction accuracy, mitigates bias, ensures consistent predictions, and improves robustness across various datasets. 1 Introduction Developing robust image classification models that generalize effectively to unseen or out-of-distribution data remains a challenging problem in computer vision. This issue largely arises from biases and limited diversity in training datasets Torralba and Efros (2011) . Standard models trained on such data often prioritize irrelevant background or contextual cues over the discriminative visual features that define each class Ribeiro et al. (2016) . Consequently, these models struggle to generalize to unfamiliar or atypical examples, undermining their reliability and practical utility in real-world applications. Learning robust joint representations for vision and language is an important challenge in modern deep learning research, where the goal is to construct a function f(\mathbf{V},\mathbf{L}) that aligns visual data \mathbf{V} and linguistic data \mathbf{L} into a unified representation capturing shared semantics while preserving modality-specific details; mathematically, this can be expressed as f:\mathcal{V}\times\mathcal{L}\to\mathcal{Z} , where \mathcal{Z} denotes the joint latent space encoding these semantics, with the primary challenge being to construct f such that it is both expressive and generalizable across diverse input types. While humans seamlessly combine information across modalities to make complex inferences (e.g., understanding product reviews by interpreting customer-uploaded images alongside customer written feedback describing product experiences), traditional machine learning approaches often process each modality in isolation, making it challenging to exploit the rich complementary information inherent across modalities. Vision-Language Models (VLMs) Radford et al. (2021) ; Jia et al. (2021) ; Li et al. (2022b) address this limitation by training a unified model on large-scale image-text pairs, enabling the formation of rich cross-modal semantic representations. These learned associations subsequently drive performance across diverse downstream applications, including image captioning, visual question answering, content moderation, context-sensitive retrieval, and multimodal sentiment analysis Du et al. (2022) . Although pretrained VLMs excel at zero-shot classification—enabling inference on new data without additional training, a promising research direction lies in leveraging these large-scale pretrained models to guide smaller task-specific classifiers tailored to particular applications. A seminal effort in this area is the GALS framework Petryk et al. (2022) , which proposes using language-based guidance from VLMs to steer task-specific classifiers through the generation of reference or guiding attention maps. Specifically, GALS transforms conventional classification labels into descriptive textual prompts, constructing semantically rich image-text pairs compatible with the VLM’s multimodal processing pipeline. These pairs are used to extract class-focused image attention maps, which highlight image regions semantically aligned with the corresponding linguistic descriptions. With these VLM-generated attention maps, a regularization mechanism is designed for the task-specific classifier, constraining its focus to align with semantically meaningful regions identified through language-guided reasoning. This approach enhances the interpretability, robustness, and accuracy of image classifiers by clarifying visual-textual relationships and reducing reliance on irrelevant visual cues. While GALS improves classification accuracy by integrating pretrained VLMs as a guidance mechanism, it inherits a fundamental limitation shared by most conventional VLMs: their reliance on deterministic embeddings. Such embeddings struggle to account for the inherent ambiguities and uncertainties in multimodal data Blundell et al. (2015) ; Oh et al. (2019) . Specifically, this framework overlooks the fact that a single textual description can correspond to multiple distinct images, just as a single image may be accurately described by various textual expressions. This constraint hinders model effectiveness, particularly in tasks requiring fine-grained distinctions or generalization across diverse data distributions. To address these challenges, probabilistic VLMs have been developed Upadhyay et al. (2023) ; Baumann et al. (2024) , focusing on uncertainty estimation. Probabilistic embeddings provide a more nuanced representation by representing multimodal data as probability distributions rather than fixed points in the embedding space. Explicitly modelling uncertainty enables these frameworks to capture the variability in visual-textual relationships, leading to significant advancements in fine-grained classification, active learning, domain adaptation, and targeted model refinement via uncertainty quantification. To accommodate multivaluedness and the ill-posed nature inherent to cross modal mappings, we introduce PARIC : “a probabilistic framework for guiding visual attention using language specifications" . PARIC leverages probabilistic VLMs to generate uncertainty-aware attention maps from descriptive textual prompts derived from classification labels. These probabilistic attention maps explicitly guide a separate task-specific neural classifier to focus on semantically meaningful visual regions, aiding model interpretability, robustness, and generalization. Building upon GALS Petryk et al. (2022) and probabilistic adapters Upadhyay et al. (2023) , our approach bridges the gap between deterministic and probabilistic multimodal paradigms. We demonstrate improvements in accuracy, robustness, and interpretability across benchmark visual classification datasets, underscoring the efficacy of uncertainty-aware multimodal guidance. 2 Background Vision-Language Models (VLMs) have been pivotal in multimodal research, enabling simultaneous processing of images and text within a unified architecture. Foundational models like CLIP Radford et al. (2021) and ALIGN Jia et al. (2021) employ contrastive learning on large-scale datasets (e.g., LAION-400M Schuhmann et al. (2021) ), to align image-text pairs in a shared embedding space, excelling in tasks such as cross-modal retrieval and zero-shot classification Wang et al. (2024) . Unified pre-training approaches such as BLIP Li et al. (2022b) , combine contrastive and generative objectives, enhancing performance in tasks like image captioning and visual question answering. However, a notable limitation of these models lies in their reliance on deterministic embeddings. Although suitable for many downstream applications, deterministic representations may not adequately address the uncertainties or variations inherent in real-world data, especially when dealing with fine-grained distinctions, unseen categories, or tasks that demand high precision and adaptability Yang et al. (2021) . Probabilistic Embeddings have recently been explored in various workflows Upadhyay et al. (2023) ; Chun et al. (2021) ; Li et al. (2022a) ; Neculai et al. (2022) , and hold the potential to address the inherent ambiguity and multivaluedness in VLMs by explicitly modeling uncertainty, offering a more nuanced representation of complex multimodal data Blundell et al. (2015) ; Oh et al. (2019) . However, training probabilistic VLMs from scratch demands substantial computational resources and large scale datasets Stirn et al. (2023) . Recent advancements address this limitation through post-hoc probabilistic methods. Specifically, Bayesian posterior approximation, as seen in Baumann et al. (2024) , estimates uncertainty over the final VLM layers without retraining. Alternatively, Upadhyay et al. (2023) introduces lightweight probabilistic adapters to convert deterministic embeddings into probabilistic embeddings, offering a computationally efficient approach that leverages pre-trained models. Language as a Guide for Visual Models based on attention mechanisms allow models to focus selectively on the most salient parts of an input, such as specific regions in an image Gan et al. (2024)

Read the complete text