Vision-Language Models display a strong gender bias

Aiswarya Konavoor, Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat · 2025

Read Vision-Language Models display a strong gender bias by Aiswarya Konavoor, Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11262.txt.

Introduction

Aiswarya Konavoor and co-authors show that vision-language models carry strong gender bias, in a 2025 paper. Such models align images and text in a shared space. That skill powers search and zero-shot labeling. It can also smuggle in social stereotypes that accuracy scores hide.

The team built a set of face photos balanced by perceived gender. They paired them with short statements about jobs and daily tasks. Statements spanned emotional, domestic, technical, and physical labor, among other fields. The model's placements of faces near phrases revealed gendered patterns. Those patterns persisted across many statement types. The study gives auditors a concrete test for bias. Model builders gain a warning to check fairness, not just accuracy.

Catalog record

CREATORAiswarya Konavoor, Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11262

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision-Language Models display a strong gender bias Aiswarya Konavoor, Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat Abstract Vision-language models (VLM) align images and text in a shared representation space that is useful for retrieval and zero-shot transfer. Yet, this alignment can encode and amplify social stereotypes in subtle ways that are not obvious from standard accuracy metrics. In this study, we test whether the contrastive vision-language encoder exhibits gender-linked associations when it places embeddings of face images near embeddings of short phrases that describe occupations and activities. We assemble a dataset of 220 face photographs split by perceived binary gender and a set of 150 unique statements distributed across six categories covering emotional labor, cognitive labor, domestic labor, technical labor, professional roles, and physical labor. We compute unit-norm image embeddings for every face and unit-norm text embeddings for every statement, then define a statement-level association score as the difference between the mean cosine similarity to the male set and the mean cosine similarity to the female set, where positive values indicate stronger association with the male set and negative values indicate stronger association with the female set. We attach bootstrap confidence intervals by resampling images within each gender group, aggregate by category with a separate bootstrap over statements, and run a label-swap null model that estimates the level of mean absolute association we would expect if no gender structure were present. The outcome is a statement-wise and category-wise map of gender associations in a contrastive vision-language space, accompanied by uncertainty, simple sanity checks, and a robust gender bias evaluation framework. 1 Introduction Vision–language models (VLMs) learn a shared representation space for images and text, enabling open-vocabulary recognition, retrieval, and zero-shot transfer [ 1 ] [ 2 ] [ 3 ] . This alignment is powerful for downstream tasks but can also carry forward and even amplify patterns present in large-scale web data, including social stereotypes that are not captured by standard accuracy metrics [ 4 ] . Prior work in both language and vision has shown that learned representations can encode human-like associations [ 2 ] [ 1 ] and that dataset composition and representation choices can affect demographic outcomes. These insights point to the importance of systematically measuring demographic associations in large multimodal models trained on loosely curated internet data [ 5 ] . In this paper, we examine whether a contrastive VLM [ 6 ] [ 7 ] [ 8 ] associates short, neutral phrases describing occupations and everyday activities more strongly with one binary gender than the other when compared against galleries of face images (Figure 1 ). This provides an interpretable way to test whether the learned geometry consistently places certain statements closer to male or female faces. We use a CLIP-style dual encoder [ 9 ] [ 10 ] , which produces unit-norm embeddings [ 9 ] [ 7 ] for both modalities, enabling stable cosine similarity comparisons while avoiding variability from decoding or sampling [ 11 ] . Our measurement design uses two components that can be independently inspected: a gallery of face images [ 12 ] split into perceived male and perceived female, and a set of short, neutral statements spanning work and activity domains. For each statement, we calculate the difference between its mean cosine similarity to male faces and female faces, with positive values indicating a stronger association with male faces and negative values with female faces [ 13 ] . We optionally average across neutral prompt templates to reduce phrasing sensitivity. All vectors are L2-normalized and evaluated with gradients disabled, producing a statement-level association score [ 14 ] that can be aggregated into broader categories such as emotional labor, cognitive labor, domestic labor, technical labor, professional roles, and physical labor. Figure 1 : Geometric illustration of association in the shared embedding space. The image vectors for a female face and a male face and the text vectors for ”Cooking” and ”CEO” are shown as unit directions, where a smaller angle implies a larger cosine similarity, and since the ”Cooking” vector is closer to the female image than to the male image we have \beta<\alpha . While the ”CEO” vector is closer to the male image than to the female image, \gamma<\Delta , which questions whether the VLMs associate statement ”Cooking” more with female faces and statement ”CEO” more with male faces. We attach confidence intervals to each score using a bootstrap procedure and also compute a label-swap null to estimate the expected bias magnitude in the absence of real gender structure [ 3 ] [ 15 ] . This guards against spurious interpretations and provides a baseline for significance [ 16 ] [ 17 ] [ 2 ] [ 18 ] . Our pipeline is lightweight and scalable, allowing evaluation across different encoders and model families. Although our main experiments use a binary gender partition, the method extends to other demographic dimensions when balanced and ethically sourced probe sets are available. The contributions of this work are 3-fold. First, we present a transparent and reproducible method to measure gender-linked associations in vision–language embedding spaces using cosine similarity alone, isolating encoder-level effects. Second, we include uncertainty estimation and null calibration to distinguish real effects from noise. Third, we introduce category-level aggregation to reveal domain-wide patterns without losing statement-level detail. The overall goal is to provide a compact, interpretable, and extensible measurement framework that practitioners can apply to understand how an encoder’s learned geometry relates to socially relevant categories. 2 Methodology 2.1 Dataset The image dataset comprised 220 face photographs [ 19 ] evenly split between perceived male and perceived female categories. All images were converted to RGB, cropped to focus on the face, and balanced in number to remove asymmetry [ 20 ] [ 21 ] . The text dataset contained 150 short, gender-neutral statements spanning six broad labor categories: emotional labor, cognitive labor, domestic labor, technical labor, professional roles, and physical labor. Each category included a diverse set of statements to allow both statement-level and category-level analysis. Figure 2 : Example face images used in the study. Representative subset of the balanced male and female face galleries used for bias measurement. Each image is cropped to focus on the face, and individuals are labeled only by perceived binary gender (Male 1–4, Female 1–4) for clarity. These samples illustrate the diversity of appearance within each group while excluding contextual elements that could confound similarity measurements. 2.2 Model and Preprocessing We used pre-trained CLIP-style contrastive encoders [ 22 ] , which provide unit-norm embeddings for both images and text, making cosine similarity a natural and stable measure [ 23 ] [ 24 ] [ 25 ] (Table 1 ). Images were resized, center-cropped, and normalized using the model’s preprocessing pipeline before being encoded into embeddings [ 26 ] . Text statements were tokenized, encoded, and L2-normalized in the same manner [ 27 ] . To reduce sensitivity to prompt wording, each statement was expanded into several neutral templates (e.g., “A person performing {x}”, “An occupation that involves {x}”), and the resulting embeddings were averaged to produce a single text vector per statement [ 3 ] [ 13 ] . Table 1 : Vision–Language Models evaluated. All are CLIP-style dual encoders. Model Architecture

Read the complete text