Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos

Haodong Chen, Qiang Huang, Jiaqi Zhao, Qiuping Jiang, Xiaojun Chang, Jun Yu · 2026

Read Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos by Haodong Chen, Qiang Huang, Jiaqi Zhao, Qiuping Jiang, Xiaojun Chang, Jun Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06931.txt.

Introduction

Haodong Chen and co-authors published a 2026 arXiv preprint. The preprint covers measuring social bias in Vision-Language models with Face-Only counterfactuals from real photos. It appeared in the arXiv PAPERS collection for 2026. Vision-Language Models (VLMs) are increasingly deployed in socially consequential settings, raising concerns about social bias driven by demographic cues.

A central challenge in measuring such social bias is attribution under visual confounding: real-world images entangle race and gender with correlated factors such. As background and clothing, obscuring attribution. The preprint gives researchers a focused account of the problem and the approach the authors take.

Catalog record

CREATORHaodong Chen, Qiang Huang, Jiaqi Zhao, Qiuping Jiang, Xiaojun Chang, Jun Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06931

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos Haodong Chen, Qiang Huang, Jiaqi Zhao, Qiuping Jiang, Xiaojun Chang, Jun Yu Abstract Vision-Language Models (VLMs) are increasingly deployed in socially consequential settings, raising concerns about social bias driven by demographic cues. A central challenge in measuring such social bias is attribution under visual confounding: real-world images entangle race and gender with correlated factors such as background and clothing, obscuring attribution. We propose a face-only counterfactual evaluation paradigm that isolates demographic effects while preserving real-image realism. Starting from real photographs, we generate counterfactual variants by editing only facial attributes related to race and gender, keeping all other visual factors fixed. Based on this paradigm, we construct FOCUS , a dataset of 480 scene-matched counterfactual images across six occupations and ten demographic groups, and propose REFLECT , a benchmark comprising three decision-oriented tasks: two-alternative forced choice, multiple-choice socioeconomic inference, and numeric salary recommendation. Experiments on five state-of-the-art VLMs reveal that demographic disparities persist under strict visual control and vary substantially across task formulations. These findings underscore the necessity of controlled, counterfactual audits and highlight task design as a critical factor in evaluating social bias in multimodal models. 1 Introduction Vision-Language Models (VLMs) are increasingly deployed in high-stakes, people-facing applications that involve explicit or implicit judgments about individuals ( 15 ; 22 ) . In practice, such judgments often manifest as ranking, screening, or assessment decisions that inform downstream actions, including hiring and candidate screening, educational allocation, socioeconomic evaluation, and trust-related decisions in safety-critical settings ( 2 ) . As VLMs become embedded in these workflows, concerns about their sensitivity to demographic cues and the resulting social bias have grown correspondingly. VisBias: real photos, context varies FOCUS: same scene, face-only edits Figure 1: FOCUS isolate facial demographic cues while keeping background, clothing, pose, and lighting fixed. Crucially, even when prompts do not explicitly reference protected attributes such as race or gender, demographic cues conveyed through facial appearance can still shape model inferences and recommendations ( 14 ; 28 ) . Such sensitivity can give rise to social bias : systematic differences in model outputs across demographic groups (here, race and gender) under matched task conditions. When these disparities are driven by demographic cues rather than decision-relevant evidence, they can produce hidden and involuntary disadvantages for certain groups, leading to disparate treatment or disparate impact in real-world deployments ( 27 ; 23 ) . Accordingly, developing reliable methods to benchmark social bias in VLMs is an increasingly urgent challenge. A central difficulty lies in attribution under visual confounding ( 25 ) . Real-world photographs entangle many correlated factors: background, clothing, pose, lighting, image quality, and scene semantics, that may co-vary with demographic attributes in uncontrolled ways ( 9 ) . Thus, disparities observed across demographic groups are difficult to interpret: they may reflect genuine sensitivity to demographic cues, or instead arise from spurious correlations in contextual features. Existing benchmarks face a persistent trade-off. Datasets built from real photos are natural but often under-controlled, whereas fully synthetic or heavily generated benchmarks allow tighter control but may deviate from real-image distributions or inherit artifacts and biases from the generator itself ( 24 ; 9 ) . This tension motivates the need for an evaluation paradigm that is simultaneously realistic and strictly controlled. To address this gap, we first construct FOCUS , a real-photo F ace- O nly C ounterfact U al S dataset. FOCUS comprises a scene-matched counterfactual image set created by editing only facial attributes associated with protected demographics–race and gender in this work–all non-demographic factors fixed, including background, clothing, pose, lighting, and camera framing (Figure 1 ). By holding the surrounding visual context constant, FOCUS isolates the causal effect of facial demographic cues from spurious scene-level correlations. Second, we introduce REFLECT , a RE al-photo F ace-on L y E dits for C ounterfac T uals benchmark for decision-oriented bias evaluation. REFLECT provides a standardized evaluation suite spanning three complementary task families: (i) Two-Alternative Forced Choice (2AFC), which elicits relative preferences between paired counterfactual variants from the same source photo; (ii) Multiple-Choice Questions (MCQ), which probe single-image categorical assessments such as salary band and education level; and (iii) Salary Recommendation, which evaluates continuous numeric decisions by asking models to recommend an annual salary given a candidate portrait and a controlled biography. Together, these tasks capture complementary bias signals across comparative judgments, categorical inference, and quantitative decision-making under strict visual control. Across experiments on five state-of-the-art VLMs, we find that demographic disparities persist even under counterfactual control , and that both their magnitude and direction vary substantially across tasks and scenarios. This variability is consequential for auditing: a model that appears benign under one format may exhibit pronounced disparities under another, particularly when outputs shift from categorical judgments to comparative choices or numeric recommendations. Overall, our findings underscore the importance of controlled, counterfactual evaluations for auditing multimodal systems deployed in socially consequential settings. Our contributions are fourfold: • We propose a controlled evaluation paradigm for measuring social bias in VLMs using face-only counterfactuals from real photographs , enabling cleaner attribution by fixing background, clothing, and other non-demographic factors while varying only race and gender. • We construct FOCUS , a real-photo face-only counterfactual dataset covering six occupations and ten race-gender groups, comprising 480 images generated with a unified editing prompt and validated through a rigorous quality-control pipeline. • We introduce REFLECT , a decision-oriented benchmark suite with three complementary task families: 2AFC (comparative judgments), MCQ (categorical assessments), and Salary Recommendation (numeric decisions), to probe bias signals across distinct input-output formats under strict visual control. • We present a systematic evaluation of five state-of-the-art VLMs, demonstrating that demographic disparities persist even under counterfactual control and that both their magnitude and direction depend strongly on task formulation and scenario. 2 Related Work Figure 2: Overview of REFLECT with FOCUS dataset construction. Starting from real photos, we generate scene-matched counterfactuals by editing only facial demographic cues while keeping all other context fixed. Using these controlled images, we evaluate VLMs with three decision-oriented tasks: (1) 2AFC , head-to-head comparisons between paired counterfactuals from the same source photo; (2) MCQ , single-image categorical judgments; and (3) Salary Recommendation , numeric salary outputs conditioned on a portrait and a standardized biography. Bias Benchmarks for LLMs A substantial body of work evaluates social bias in LLMs by testing whether model behavior varies in response to demographic cues. Classic benchmarks probe preferences between stereotypical and anti-stereotypical alternatives ( 17 ; 18 ) , ambiguity-sensitive QA designed to surface stereotype-driven defaults ( 21 ) , and harms in open-ended generation such as toxicity or biased portrayals ( 10 ; 6 ; 4 ) . In decision-oriented settings, 19 study disparities in employment and salary recommendations by injecting demographic signals via names and resume-like text. These benchmarks establish core paradigms for bias elicitation, but operate primarily in text-only settings. Bias Benchmarks for VLMs As foundation models become multimodal, concerns about social bias extend to VLMs, where demographic cues can arise from both text and visual appearance. Prior work adapts LLM-style stereotyping probes to multimodal inputs ( 30 ) , while VisBias evaluates explicit and implicit bias using in-the-wild images and diverse elicitation

Read the complete text