Unveiling the “Fairness Seesaw”: Discovering and Mitigating Gender and Race Bias in Vision-Language Models

Jian Lan 1,2 * Udo Schlegel 1,2 Gengyuan Zhang 1,2 Tanveer Hannan 1,2 Haokun Chen 1 Thomas Seidl 1,2 LMU Munich 1 Munich Center for Machine Learning (MCML) 2 {lan}@dbs.ifi.lmu.de * Corresponding author · 2025

Read Unveiling the “Fairness Seesaw”: Discovering and Mitigating Gender and Race Bias in Vision-Language Models by Jian Lan 1,2 * Udo Schlegel 1,2 Gengyuan Zhang 1,2 Tanveer Hannan 1,2 Haokun Chen 1 Thomas Seidl 1,2 LMU Munich 1 Munich Center for Machine Learning (MCML) 2 {lan}@dbs.ifi.lmu.de * Corresponding author (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23798.txt.

Introduction

Jian Lan and co-authors uncovered a fairness seesaw in vision-language models in a 2025 paper. These models pair images with text and now serve many public uses. Their social biases, though, are poorly understood, hidden inside opaque training. The team asked how strongly the models show gender and race bias in what they generate.

Their systematic hunt found that fixing bias on one axis can worsen it on another, like pressing one end of a seesaw. They trace where in the model the biases sit and propose mitigation steps. The seesaw finding warns against one-dimensional fairness checks. Auditors get a method to measure several biases jointly before a model is released on the public.

Catalog record

CREATORJian Lan 1,2 * Udo Schlegel 1,2 Gengyuan Zhang 1,2 Tanveer Hannan 1,2 Haokun Chen 1 Thomas Seidl 1,2 LMU Munich 1 Munich Center for Machine Learning (MCML) 2 {lan}@dbs.ifi.lmu.de * Corresponding author
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23798

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unveiling the “Fairness Seesaw”: Discovering and Mitigating Gender and Race Bias in Vision-Language Models Jian Lan 1,2 * Udo Schlegel 1,2 Gengyuan Zhang 1,2 Tanveer Hannan 1,2 Haokun Chen 1 Thomas Seidl 1,2 LMU Munich 1 Munich Center for Machine Learning (MCML) 2 {lan}@dbs.ifi.lmu.de * Corresponding author Abstract . Although Vision-Language Models (VLMs) have achieved remarkable success, the knowledge mechanisms underlying their social biases remain a "black box", where fairness- and ethics-related problems harm certain groups of people in society. It is unknown to what extent VLMs yield gender and race bias in generative responses. In this paper, we conduct a systematic discovery of gender and race bias in state-of-the-art VLMs, focusing not only on surface-level responses but also on the internal probability distributions and hidden state dynamics. Our empirical analysis reveals three critical findings: 1) The Fairness Paradox : Models often generate "fair" text labels while maintaining highly skewed confidence scores (mis-calibration) toward specific social groups. 2) Layer-wise Fluctuation : Fairness knowledge is not uniformly distributed; it peaks in intermediate layers and undergoes substantial "knowledge erosion" in the final layers. 3) Residual Discrepancy : Within a single hidden layer, different residual streams carry conflicting social knowledge—some reinforcing fairness while others amplifying bias. Leveraging these insights, we propose RES-FAIR ( RES idual F low A djustment for I nference R ecalibration), a post-hoc framework that mitigates bias by localizing and projecting hidden states away from biased residual directions while amplifying fair components. Evaluations on PAIRS and SocialCounterfactuals datasets demonstrate that our discovery-based approach significantly improves response fairness and confidence calibration without compromising general reasoning abilities. Our work provides a new lens for understanding how multi-modal models store and process sensitive social information. CCS: Mathematics of computing Probabilistic representations CCS: Social and professional topics Gender CCS: Social and professional topics Race and ethnicity 1. Introduction To strengthen gender and race fairness in Vision Language Models (VLMs), models should avoid exhibiting stereotypes such as gender bias or race bias when interacting with humans. Such fairness settings follow the previous work ( Fraser and Kiritchenko, 2024 ; Howard et al., 2024 ; Gerych et al., 2024 ) . We use the term “social category” to indicate an overall social class, such as “gender” or “race”, and use “social attributes” to indicate different group labels in a class (e.g., “male” and “female” are two attributes in gender). Addressing gender and race bias in VLMs has gained increasing attention. Gerych et al. (2024) and Berg et al. (2022) study image retrieval ( Lahajal and S, 2024 ) , measuring how often an attribute is correctly retrieved. However, such frequency-based embedding matches cannot capture VLMs’ real generative behavior, leaving biases in free-form responses underexplored. The open-ended nature of generation further complicates evaluation. A common workaround is multiple-choice selection (MCS) ( Santurkar et al., 2023 ) , which constrains the response space and enables accuracy and probability-based analysis. While MCS has been widely used in text generation ( Wang et al., 2024c ; Chen et al., 2024 ) , Fraser and Kiritchenko (2024) first applied it to study gender and race bias in VLMs, prompting models to classify a person in an image from candidate options. However, focusing solely on specific images is limiting, as a single person does not represent broader attributes (see Fig. 1 , left). For example, if a model predicts “pilot” correctly for both a man and a woman, it may appear gender-fair, yet this overlooks the background knowledge underlying bias. Single-sample evaluations can thus mislead: classifying one female correctly does not imply equal treatment of women as a group. This motivates a different setting where prompts target general social categories (e.g., gender preferences, Fig. 1 ), with images as references but not the sole basis for responses. Here, models reveal their internal bias directly: a biased model may claim males are better computer scientists, while a fair one answers that both genders are the same. Figure 1. A comparison showing previous evaluation problems on the left and improved MCS inputs with model generations on the right. A comparison showing previous evaluation problems on the left and improved MCS inputs with model generations on the right. Further, previous works ( Gerych et al., 2024 ; Berg et al., 2022 ; Howard et al., 2024 ; Huang et al., 2025 ) only study models’ responses or embedding similarity, leaving models’ confidence levels towards different attributes under-explored. Motivated by recent work ( Lan et al., 2025a ; Lan et al., 2025b ) , which highlights the importance of VLMs’ probability distribution (also known as model confidence), we study not only model responses but also confidence levels. We believe a fair enough model should generate fair responses while having a reliable confidence as shown in Figure 1 . This work aims to answer the following research questions: RQ1 : To what extent do SOTA VLMs yield gender and race bias in responses? RQ2 : Do VLMs have reliable confidence levels in different social groups? RQ3 : If VLMs have bias in responses and confidence, how can we mitigate these? To answer these questions, we start by examining how well VLMs generate fair responses with fair confidence on two datasets: PAIRS ( Fraser and Kiritchenko, 2024 ) and SocialCounterfactuals (SCF) ( Howard et al., 2024 ) . We follow the previous work ( Fraser and Kiritchenko, 2024 ) using the MCS task. Differently, due to the lack of well-designed prompts that support evaluating models’ bias in the context of gender and race bias, we construct new prompts and modify the data labels based on PAIRS and SCF, and change the way models make choices, as detailed in section 3 . We observe that even the latest SOTA VLMs, such as LLaVA-NeXT-13B ( Liu et al., 2024b ) and Qwen2.5-VL-32B ( Bai et al., 2025 ) , still struggle to produce fair responses, with probability distributions also mis-calibrated. Analysis shows that fairness peaks in intermediate layers but fluctuates in the final ten layers, and residuals within a layer do not consistently help—some even amplify bias. We term this phenomenon the “Fairness Seesaw” : a delicate and often unstable trade-off where the model’s internal representations fluctuate between fairness-promoting and bias-reinforcing states across different layers and residual streams. Much like a seesaw, a slight adjustment in one component can lead to a significant tilt in the model’s overall social bias, making consistent fairness difficult to achieve without precise intervention. Motivated by this, we propose RES-FAIR ( RES idual F low A djustment for I nference R ecalibration), a post-hoc method to disentangle residuals into fair and biased components. Starting from the last n^{th} layer, we evaluate each residual’s effect on responses, categorize them as fair or biased, and compute their mean vectors. We then apply an orthogonal projection to adjust hidden states toward fairness, as detailed in Section 4 . Our key contributions are : 1) To the best of our knowledge, we are the first to study VLMs’ confidence levels on gender and race bias and reveal that even SOTA VLMs still have difficulty generating absolute fair contents. 2) To better support studying VLMs’ generative responses, we construct and contribute new prompts and MCS task based on current dataset

Read the complete text