PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection
Mahdiyar Molahasani, Azadeh Motamedi, Michael Greenspan, Il-Min Kim, Ali Etemad · 2025
Read PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection by Mahdiyar Molahasani, Azadeh Motamedi, Michael Greenspan, Il-Min Kim, Ali Etemad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08979.txt.
Introduction
Mahdiyar Molahasani, Azadeh Motamedi, Michael Greenspan, Il-Min Kim, and Ali Etemad present PRISM, a way to strip hidden bias from vision-language models without new data. Their paper from twenty twenty-five notes that models like CLIP inherit skewed links from their training sets. Past fixes need set bias classes or extra data. PRISM needs neither, and it works across tasks.
First, a language model writes scene texts from plain class prompts, surfacing the spurious links in words. Then a contrast-style loss learns a projection of the embeddings. The projection trims those links while keeping image-text alignment intact. Tests on the standard Waterbirds and CelebA sets beat current debiasing methods. Code is public. The authors stress the data-free stance: bias can be cut with the model's own space alone.
Catalog record
| CREATOR | Mahdiyar Molahasani, Azadeh Motamedi, Michael Greenspan, Il-Min Kim, Ali Etemad |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08979
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection Mahdiyar Molahasani, Azadeh Motamedi, Michael Greenspan, Il-Min Kim, Ali Etemad Abstract We introduce Projection-based Reduction of Implicit Spurious bias in vision-language Models (PRISM), a new data-free and task-agnostic solution for bias mitigation in VLMs like CLIP. VLMs often inherit and amplify biases in their training data, leading to skewed predictions. PRISM is designed to debias VLMs without relying on predefined bias categories or additional external data. It operates in two stages: first, an LLM is prompted with simple class prompts to generate scene descriptions that contain spurious correlations. Next, PRISM uses our novel contrastive-style debiasing loss to learn a projection that maps the embeddings onto a latent space that minimizes spurious correlations while preserving the alignment between image and text embeddings. Extensive experiments demonstrate that PRISM outperforms current debiasing methods on the commonly used Waterbirds and CelebA datasets We make our code public at: https://github.com/MahdiyarMM/PRISM . 1 Introduction Vision-Language Models (VLMs) have emerged as a class of machine learning models capable of understanding and generating multi-modal content by jointly processing images and textual descriptions. Prominent VLMs, such as Contrastive Language-Image Pre-training (CLIP) [ 20 ] , have demonstrated impressive performance in zero-shot classification and cross-modal retrieval. By aligning textual and visual representations in a shared embedding space, these models enable seamless interactions between vision and language tasks. Despite their unprecedented effectiveness, VLMs inherit and propagate systemic biases from their large-scale pretraining data, leading to inaccurate or skewed predictions. (a) Waterbirds. (b) CelebA. Figure 1 : Performance comparison among data-free CLIP debiasing methods in terms of debiasing performance (i.e., the accuracy of the worst group) vs. the overall accuracy on (a) Waterbirds and (b) CelebA datasets. Our proposed method, PRISM, achieves state-of-the-art debiasing performance (higher worst-group accuracy) without compromising overall accuracy. Biases in VLMs manifest in various forms, including social [ 33 , 19 , 30 , 27 ] and contextual biases [ 31 , 26 ] , often influenced by ‘spurious correlations’ in training data. Spurious correlations can stem from associating non-target attributes, such as image backgrounds or contextual features, with labels in unintended ways [ 5 ] . For instance, CLIP’s zero-shot classifiers frequently associate ‘camel’ with ‘desert’ and ‘panda’ with ‘bamboo’, reflecting underlying biases in training data. These biases can have real-world implications reducing model robustness in diverse settings. Therefore, mitigating implicit bias in VLMs has become a critical research direction. Recent research has investigated various strategies to identify , quantify , and reduce spurious correlations in VLMs. These approaches can be classified into three categories: feature-space transformations [ 7 , 29 ] , prompt-based debiasing [ 5 , 6 , 1 ] , and adapter design [ 17 ] . Methods with a primary focus on the feature space [ 7 , 29 ] attempt to transform feature embeddings to eliminate spurious correlations. Prompt-based techniques [ 5 , 6 , 1 ] aim to redefine textual inputs to mitigate bias but often struggle with generalization. Meanwhile, adapter-based approaches such as [ 17 ] address implicit bias in VLMs by modifying representations at various stages of the model, including but not limited to the final layer. Despite the effectiveness of these approaches in addressing spurious bias in VLMs, they fall short in many instances. These methods mostly require fixed predefined bias categories makes them difficult to scale to task-agnostic settings [ 7 , 5 , 6 ] . Furthermore, these debiasing solutions either compromise the overall performance of the model by removing essential semantic content [ 1 , 5 ] or rely on external visual data for debiasing [ 7 , 17 , 29 ] . This underscores the need for a new debiasing method without the reliance on predefined bias categories or any external data that can preserve essential semantic content without compromising overall accuracy. To address these limitations, we propose P rojection-based R eduction of I mplicit S purious bias in vision-language M odels (PRISM). PRISM consists of two key stages. First, we prompt an LLM with labels to return a list of potential spurious connections corresponding to these labels and dynamically generate biased scene disruptions based on the extracted spurious attributes. Next, we find a projection of the embedding space that can effectively remove the impact of spurious attributes using the generated biased scene descriptions. To this end, we propose a novel loss named Latent space Debiasing loss ( LD ) to learn the debiasing projection in the embedding space effectively. Our proposed LD loss is a contrastive-style debiasing loss that aligns text and image embeddings based on semantic meaning rather than spurious correlations. It ensures that the text embeddings of the same class but different spurious attributes remain similar to each other, and the text embeddings of different classes but the same spurious attribute remain distinct. By removing the impact of bias attributes from the embedding space, PRISM generates representations with no or reduced statistical dependence on the bias. To measure this dependency, previously proposed methods mostly employ a metric capable of capturing statistical dependencies, such as Mutual Information [ 22 ] , Hilbert-Schmidt Independence Criterion (HSIC) [ 4 ] , or bias scores [ 7 ] . In contrast, our proposed method eliminates the need for direct dependency measurement by utilizing an LLM to generate bias-aware descriptions. These descriptions capture common bias attributes in the dataset using only the labels. Our approach facilitates debiasing without the need for predefined bias categories, external text or image datasets, or lossy data conversions to identify bias features. This makes PRISM an effective, task-agnostic, and data-free solution for debiasing VLMs. We assess the performance of PRISM on CLIP using different benchmarks, including Waterbirds and CelebA datasets, and compare its performance against various baselines. The results highlight the capability of the proposed PRISM in mitigating spurious biases in VLMs without compromising the overall performance. Figure 1 illustrates that our proposed PRISM achieves state-of-the-art debiasing performance without compromising the total accuracy on both Waterbirds and CelebA benchmarks. Our key contributions in this paper can be summarized as follows: • We leverage LLMs to generate biased textual descriptions that reflect common spurious correlations found in datasets. This provides a dynamic and adaptable way to identify biases without relying on predefined bias categories. • We propose a novel task-agnostic data-free debiasing method for VLMs that learns a projection of the embedding space, removing the impact of spurious attributes using a novel contrastive style loss function. • We demonstrate that our proposed method achieves state-of-the-art debiasing performance without compromising the model’s overall accuracy. To allow rapid reproducibility and contribute to the community, we make our code public at: https://github.com/MahdiyarMM/PRISM . 2 Related Works Bias identification in VLMs. Recent research has placed considerable emphasis on identifying and assessing bias in foundation models [ 9 ] , including VLMs [ 2 , 25 ] . The initial step in debiasing involves recognizing and evaluating the sources of bias. Different frameworks proposed in [ 31 ] and [ 11 , 21 ] comprehensively explored various types of biases in vision and text datasets, respectively. Previous works [ 28 , 2 , 25 ] investigated bias in CLIP’s representation space and proposed different methods