Beauty and the Bias: Exploring the Impact of Attractiveness on Multimodal Large Language Models

Aditya Gulati, Moreno D’Incà, Nicu Sebe, Bruno Lepri, Nuria Oliver · 2025

Read Beauty and the Bias: Exploring the Impact of Attractiveness on Multimodal Large Language Models by Aditya Gulati, Moreno D’Incà, Nicu Sebe, Bruno Lepri, Nuria Oliver (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16104.txt.

Introduction

People judge attractive faces more kindly, a bias called the halo effect, and chatbots that see images may share it. Aditya Gulati and co-authors tested that risk in this 2025 study. They used seven open multimodal language models, which read both pictures and text. Each model judged social cases paired with face photos.

The set held ninety-one cases on topics such as hiring and lending, and photos of 462 people. Each face appeared both plain and with a beauty filter applied. Looks changed the models' calls in most cases, at a rate above eighty-six percent on average. The authors name this an attractiveness bias in the models. The finding warns builders of hiring and credit tools that face photos can sway an AI judge.

Catalog record

CREATORAditya Gulati, Moreno D’Incà, Nicu Sebe, Bruno Lepri, Nuria Oliver
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16104

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beauty and the Bias: Exploring the Impact of Attractiveness on Multimodal Large Language Models Aditya Gulati, Moreno D’Incà, Nicu Sebe, Bruno Lepri, Nuria Oliver Abstract Physical attractiveness matters. It has been shown to influence human perception and decision-making, often leading to biased judgments that favor those deemed attractive in what is referred to as the “attractiveness halo effect”. While extensively studied in human judgments in a broad set of domains, including hiring, judicial sentencing or credit granting, the role that attractiveness plays in the assessments and decisions made by multimodal large language models (MLLMs) is unknown. To address this gap, we conduct an empirical study with 7 diverse open-source MLLMs evaluated on 91 socially relevant scenarios and a diverse dataset of 924 face images –corresponding to 462 individuals both with and without beauty filters applied to them. Our analysis reveals that attractiveness impacts the decisions made by MLLMs in 86.2% of the scenarios on average, demonstrating substantial bias in model behavior in what we refer to as an attractiveness bias . Similarly to humans, we find empirical evidence of the existence of the attractiveness halo effect in 94.8% of the relevant scenarios: attractive individuals are more likely to be attributed positive traits, such as trustworthiness or confidence, by MLLMs than unattractive individuals. Furthermore, we uncover gender, age and race biases in a significant portion of the scenarios which are also impacted by attractiveness, particularly in the case of gender, highlighting the intersectional nature of the algorithmic attractiveness bias. Our findings suggest that societal stereotypes and cultural norms intersect with perceptions of attractiveness in MLLMs in a complex manner. Our work emphasizes the need to account for intersectionality in algorithmic bias detection and mitigation efforts and underscores the challenges of addressing biases in modern MLLMs. Keywords Cognitive Biases, Attractiveness Halo Effect, Multimodal Large Language Models, Bias Mitigation 1 Introduction Figure 1: Overview of the adopted experimental methodology to study the existence of an attractiveness bias in multimodal large language models (MLLMs). (A) As facial stimuli, we use a diverse set of face images of 462 distinct individuals ( \mathcal{X}^{o} ) and their corresponding beautified versions ( \mathcal{X}^{b} ) after applying a beauty filter, which enables to control for attractiveness. We define 91 scenarios in socially relevant areas (jobs, traits and conditions) consisting of pairs of words. (B) For each scenario, the MLLM is provided as input a face image and a textual prompt with a question about the person in the image which has two possible answers (a) or (b). (C) To measure the existence of an attractiveness bias, we compare the distributions of the answers provided by the MLLM when prompted with \mathcal{X}^{o} vs \mathcal{X}^{b} . A reliance on attractiveness by the model would lead to statistically significant differences in the answers. We run the scenarios with three different seeds to ensure robustness in the results. Physical attractiveness plays an invisible yet powerful role in human judgments and decision-making. Research since the 1970s has consistently shown that individuals deemed physically attractive are often perceived more favorably across a variety of positive traits, including intelligence [ Dion et al.(1972) , Kanazawa(2011) , Talamas(2016) , Gulati et al.(2024) ] , sociability [ Miller(1970) ] , trustworthiness [ Todorov(2008) , Gulati et al.(2024) ] , happiness [ Mathes and Kahn(1975) , Golle et al.(2013) , Gulati et al.(2024) ] or success in life [ Dion et al.(1972) ] . This phenomenon, often referred to as the attractiveness halo effect [ Dion et al.(1972) ] leads to biases that can shape outcomes in consequential areas in people’s lives, as attractive individuals are thought to be better students [ Ritts et al.(1992) ] or politicians [ Banducci et al.(2008) ] , more qualified for jobs [ Cash and Kilcullen(1985) , Hosoda et al.(2003) ] , and are more likely to receive more lenient judicial sentences [ Wilson and Rule(2015) , Wiley(1995) ] , promotions and higher salaries [ Frieze et al.(1991) , Hamermesh and Biddle(1994) ] than less attractive people. From an algorithmic perspective, the study of biases in machine learning models has gained significant attention in the past decade [ Barocas et al.(2019) , Mehrabi et al.(2021a) ] . An algorithmic bias is a systematic and unfair treatment of individuals or groups of individuals based on socially relevant characteristics, such as race, gender, age or religion. As machine learning models increasingly influence high-stakes decisions across domains, including hiring [ Raghavan et al.(2020) ] , healthcare [ Parikh et al.(2019) ] , education [ Baker and Hawn(2022) ] , social service provision [ Gillingham(2016) ] and criminal justice [ Travaini et al.(2022) ] , measuring and mitigating algorithmic bias is a priority which is reflected in existing or upcoming regulation, such as the European AI Act 1 1 1 https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng . While studied and understood in human contexts, the extent to which an attractiveness bias – i.e. , a differential treatment of individuals exclusively based on their perceived attractiveness– and the attractiveness halo effect – i.e. , the attribution of positive yet unrelated traits to individuals who are perceived as attractive– exists in algorithmic systems remains largely underexplored. With the advent and wide adoption of multimodal large language models MLLMs 2 2 2 In this paper, we use the term MLLM to refer to models that integrate text and images. [ Wu et al.(2023) , Zhang et al.(2024b) ] , algorithmic biases, including the attractiveness bias and the attractiveness halo effect, have become a growing concern. Unlike large language models (LLMs) that rely solely on textual data, MLLMs are able to process both textual and visual inputs, enabling them to interpret and generate responses based on complex multimodal information. These capabilities make MLLMs highly versatile, powerful and applicable to numerous vision-and-language tasks, ranging from image captioning, visual question answering [ Hu et al.(2024) ] and content creation 3 3 3 https://openai.com/index/sora-is-here/ , https://www.wired.com/story/linkedin-ai-generated-influencers/?utm_source=chatgpt.com to conversational AI grounded in visual context [ Pan et al.(2023) , Dong et al.(2024) ] . However, these models can inadvertently replicate or even magnify appearance-based biases such as the attractiveness bias. When MLLMs are trained on datasets containing images and descriptions of individuals, these models may implicitly treat attractiveness as a relevant factor, raising two key concerns: (1) MLLMs may exhibit an attractiveness bias , hence making decisions or judgments that differ based solely on an individual’s perceived attractiveness; and (2) MLLMs, like humans, may behave according to the attractiveness halo effect , thus favoring or assigning positive traits to individuals who are perceived as attractive, even when those traits are unrelated to the person’s actual abilities or character. In both cases, MLLMs could lead to a different or even preferential treatment of individuals who are considered to be more attractive by the model, potentially influencing critical decisions in areas such as hiring recommendations or educational assessments. Therefore, investigating the existence of an attractiveness bias and the presence and mechanisms of the attractiveness halo effect in MLLMs are essential to ensure fair outcomes in their deployment. This paper addresses these issues by conducting an empirical evaluation of seven open-source MLLMs of different sizes, listed in Table 1 . A unique aspect of our methodology is the use of beauty filters

Read the complete text