A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models
Junhyuk Choi, Minju Kim, Yeseon HongBugeun KimDepartment of Artificial Intelligence, Chung-Ang UniversitySeoul, Republic of Korea {chlwnsgur129, minjunim, ghddptjs, bgnkim}@cau.ac.kr · 2025
Read A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models by Junhyuk Choi, Minju Kim, Yeseon HongBugeun KimDepartment of Artificial Intelligence, Chung-Ang UniversitySeoul, Republic of Korea {chlwnsgur129, minjunim, ghddptjs, bgnkim}@cau.ac.kr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20901.txt.
Introduction
Junhyuk Choi, Minju Kim, Yeseon Hong, and Bugeun Kim test large vision language models for social stereotypes tied to color in a 2025 paper. Past checks of bias in such models missed two things: scores that underweight content words, and data sets that ignore color as a factor. The team build new scores based on a classic model of stereotype content.
They also release a benchmark that probes gender, race, and color stereotypes together. Testing eight models, they find the new scores capture stereotypes well, and that models show color stereotypes alongside gender and race ones. Model design and size seem to shape how strong those biases run.
Catalog record
| CREATOR | Junhyuk Choi, Minju Kim, Yeseon HongBugeun KimDepartment of Artificial Intelligence, Chung-Ang UniversitySeoul, Republic of Korea {chlwnsgur129, minjunim, ghddptjs, bgnkim}@cau.ac.kr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 113 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20901
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models Junhyuk Choi, Minju Kim, Yeseon HongBugeun KimDepartment of Artificial Intelligence, Chung-Ang UniversitySeoul, Republic of Korea {chlwnsgur129, minjunim, ghddptjs, bgnkim}@cau.ac.kr Abstract As large vision language models(LVLMs) rapidly advance, concerns about their potential to learn and generate social biases and stereotypes are increasing. Previous studies on LVLM’s stereotypes face two primary limitations: metrics that overlooked the importance of content words, and datasets that overlooked the effect of color. To address these limitations, this study introduces new evaluation metrics based on the Stereotype Content Model (SCM). We also propose BASIC, a benchmark for assessing gender, race, and color stereotypes. Using SCM metrics and BASIC, we conduct a study with eight LVLMs to discover stereotypes. As a result, we found three findings. (1) The SCM-based evaluation is effective in capturing stereotypes. (2) LVLMs exhibit color stereotypes in the output along with gender and race ones. (3) Interaction between model architecture and parameter sizes seems to affect stereotypes. We release BASIC publicly on [anonymized for review]. † † footnotetext: Equal contribution. 1 Introduction As large vision-language models (LVLMs) continue to advance, they are increasingly being utilized in perceiving visual images across diverse domains Xu et al. 2024 ; Gu et al. 2024 ; Wang et al. 2025 . However, when employing LVLMs, it is essential to acknowledge that models may be affected by stereotypes, potentially influencing the models’ perception. In humans, a stereotype refers to cognitive generalizations about the characteristics of individuals belonging to a specific social group VandenBos 2007 . So, people often judge others unconsciously based on their visual appearance such as gender, race, and color Cloutier et al. 2005 ; Eberhardt et al. 2004 ; Elliot et al. 2013 . Given that LVLMs are usually trained on human-generated data, it is likely that they have learned human stereotypes. As a result, LVLMs sometimes exhibit harmful stereotypes, leading to bad consequences Jiang et al. 2024 ; Howard et al. 2025 ; Fraser and Kiritchenko 2024 . Therefore, the need to identify such stereotypes is becoming important. Researchers recently designed various benchmarks to quantify stereotypes in LVLMs Adewumi et al. 2024 ; Wang et al. 2024a ; Wang et al. 2024b ; Sathe et al. 2024 ; Fraser and Kiritchenko 2024 . Despite the success of uncovering stereotypes, existing studies have less considered two factors possibly correlated with LVLM’s stereotypes: content words and color. First, regarding content words, we suspect that LVLMs may use positive sentiment words when generating stereotypical content. Many previous studies have relied on sentiment analysis to assess the stereotypes Wang et al. 2024a ; Raj et al. 2024 ; Gerych et al. 2024 . While this approach has contributed, sentiment-based evaluation cannot fully mirror how humans evaluate stereotypes. For instance, even positive sentences can reproduce implicit social biases toward particular social groups Howard et al. 2025 ; Siy and Cheryan 2016 . Therefore, to evaluate stereotypes more precisely, it is necessary to adopt a socio-psychological framework that captures such implicit biases in text and complements sentiment analysis. So, we adopt Stereotype Content Model (SCM), a stereotype evaluation framework used in human study Fiske et al. 2018 . Second, regarding color, LVLMs may exhibit stereotypes because of different colors in the image. Previously, researchers in vision-language model reported that model responds inconsistently due to image differences, even when images contain similar scenes Wu et al. 2025 ; Kantipudi et al. 2020 ; Liang et al. 2025 . So, studies on stereotypes have employed image pairs to control such differences. Previous attempts successfully built controlled image pairs, however, they overlooked the potential impact of color tone. As humans are influenced by color differences Elliot and Maier 2013 ; Kaya and Epps 2004 ; Wexner 1954 , we suspect that color influences stereotypes in LVLMs. To justify such suspicion, we suggest a way to evaluate whether LVLMs affected by different color tones. Thus, in this paper, we propose two metrics inspired by SCM and a new Benchmark for assessing stereotypes with Image Colors, BASIC. Additionally, we attempt to examine which part of LVLMs affect stereotypes, by comparing different model architectures. We believe that a systematic comparison between model architectures can hint at components relevant to stereotypes. As a result, this paper has the following contributions: 1. Two SCM-based metrics: Based on social psychology, we propose automatic metrics that can complement sentiment metrics. 2. BASIC benchmark: We suggest a benchmark that can separate the effect of colors from other stereotypes. 3. Systematic analysis: By comparing eight LVLMs, we found the effect of model architecture and size on stereotype is limited. 2 Related Work This section reviews existing research in terms of (1) used metrics, (2) designed benchmarks, and (3) tested LVLMs for analyzing stereotypes. The following paragraphs elaborate each of these points. First, regarding content words, prior studies have mainly assessed model responses based on sentiment polarity Inoshita and Zhou 2024 ; Hu et al. 2025 ; Gerych et al. 2024 , which limits their ability to capture the subtle meanings embedded in text. In the case of LVLMs, responses often exhibit positive sentiments because they were reinforced by human feedback Li et al. 2024 , making it difficult to detect stereotypes solely through sentiment-based metrics. Therefore, more sophisticated measures that directly target stereotypes are needed. In response to this limitation, recent work has begun incorporating social-psychological evaluation frameworks to analyze stereotypes. For instance, Howard et al. 2025 was influenced by the Stereotype Content Model (SCM). The study used the dictionary from Nicolas et al. 2021 to measure the frequency of SCM-related words in image captions. However, this approach is limited by its reliance on word frequency, without accounting for contextual or derived meanings. Second, regarding benchmarks, prior studies have emphasized the importance of constructing paired images to assess stereotypes of LVLMs. Traditional studies constructed diverse images based on common themes such as gender or race, and compared average evaluation scores across these thematic sets Mandal et al. 2023 ; Janghorbani and De Melo 2023 ; Qiu et al. 2023 . However, such methods often suffer from insufficient control over confounding variables unrelated to the targeted stereotype. For example, if gender, race, and background all change simultaneously, any observed bias cannot be attributed purely to gender when aiming to examine gender-related stereotypes. To address this limitation, recent works have proposed that changing a targeted attribute while keeping the rest of the image nearly identical ( Kim et al. 2025 ; Howard et al. 2025 ; Howard et al. 2024 ; Fraser and Kiritchenko 2024 ) . While these studies devote considerable effort to image construction, unintended color differences still appear in the generated images. Considering the visual differences can influence the model’s internal representations ( De and Pedersen 2021 ; Kantipudi et al. 2020 ; Liang et al. 2025 ) , thereby potentially affecting the manifestation of stereotypes. Thus, color differences should be considered in stereotype evaluation frameworks to provide a more accurate evaluation. Figure 1: Examples from our BASIC dataset. Top row shows chess players with different colors (white, blue, red), retaining race and gender. Bottom row shows DJs with different race or gender, retaining white color. Lastly, regarding model