Measuring Stereotype and Deviation Biases in Large Language Models

Daniel Wang, Eli Brignac, Minjia Mao, Xiao Fang · 2025

Read Measuring Stereotype and Deviation Biases in Large Language Models by Daniel Wang, Eli Brignac, Minjia Mao, Xiao Fang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06649.txt.

Introduction

Daniel Wang and co-authors test large language models in a 2025 study for two kinds of bias. One kind links traits to a group by habit, such as a faith or a party. The other kind shows in skewed mixes of people in text the models write, far from real-world shares.

The team asked four advanced models to write profiles of people and then checked the links that emerged. All four models showed both kinds of bias toward many groups. Such skew can mislead readers who trust model output about users. The study gives builders a clear way to measure the harm.

Catalog record

CREATORDaniel Wang, Eli Brignac, Minjia Mao, Xiao Fang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06649

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring Stereotype and Deviation Biases in Large Language Models Daniel Wang, Eli Brignac, Minjia Mao, Xiao Fang Abstract Large language models (LLMs) are widely applied across diverse domains, raising concerns about their limitations and potential risks. In this study, we investigate two types of bias that LLMs may display: stereotype bias and deviation bias. Stereotype bias refers to when LLMs consistently associate specific traits with a particular demographic group. Deviation bias reflects the disparity between the demographic distributions extracted from LLM-generated content and real-world demographic distributions. By asking four advanced LLMs to generate profiles of individuals, we examine the associations between each demographic group and attributes such as political affiliation, religion, and sexual orientation. Our experimental results show that all examined LLMs exhibit both significant stereotype bias and deviation bias towards multiple groups. Our findings uncover the biases that occur when LLMs infer user attributes and shed light on the potential harms of LLM-generated outputs. Key words: large language model, bias evaluation, stereotype bias, deviation bias Introduction Large language models (LLMs) are large-scale AI models trained on massive amounts of data that can process natural language instructions and generate texts across a range of applications, including healthcare, education, law, and finance [ 1 ] . Because of the wide applications, LLMs have attracted enormous interest from researchers [ 1 ] and consumers [ 2 ] in recent years. For example, LLMs can provide medical advice, support personalized learning, analyze legal documents, and facilitate financial reasoning [ 1 ] . Given the extensive applications of LLMs in real-world scenarios, understanding their limitations and potential risks is crucial. In the context of computer systems, bias is defined as when computer systems "systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others." [ 3 ] Since LLMs are trained on large-scale internet data that reflects societal inequalities and prejudices by humans, such as books, websites, and social media posts, it is possible for a model to inherit or amplify the biases of the content it is trained on. This bias is also referred to as social bias because it exhibits harm to vulnerable or minority social groups of people [ 4 ] . Existing studies have found that LLMs display bias across multiple demographic categories, including gender, ethnicity, nationality, politics, and occupation [ 5 , 6 , 7 , 8 , 9 ] . For example, Abid et al. find that GPT-3 exhibits religious bias, with prompts containing "Muslim" producing responses involving violent language [ 10 ] . Therefore, it is crucial to study and examine the bias of LLM-generated content. Existing literature on LLM-generated content bias evaluation utilizes two different evaluation principles, depending on whether a reference (or ideal) output property is given by human evaluators. On one hand, without a reference output property, LLMs can yield stereotype bias, which indicates that LLMs provide a biased and consistent output for a certain group of people. For example, Wan et al. demonstrate that AI-generated reference letters for male job candidates are more likely to include traits such as "leadership" and "ability," while female job candidates are more frequently associated with traits such as "communal" and "personal" [ 11 ] . Since LLMs are trained on corpora written by humans, the study on stereotype bias can bring insight into whether LLMs learn and amplify societal stereotypes in human society. On the other hand, with a reference output property, LLMs can exhibit deviation bias, which means the generated content does not satisfy the ideal property. For example, researchers find that news produced by LLMs tends to underestimate minority groups compared to human-written ones [ 12 ] . Deviation bias refers to whether the generated content aligns with a given reference output property, which determines whether stereotype tendencies reflect a realistic social bias against the ideal property preferred by human evaluators. Existing studies on bias evaluation primarily use either one of the evaluation principles, which may overlook important evaluation standards. For example, Shrawgi et al. assign a person from a national positive or negative stereotype [ 13 ] . However, people from different nations may exhibit different attributes. Without a ground truth distribution, it is hard to determine whether a stereotype is representative or biased. As another example, previous work evaluates the difference between election voting by LLMs and humans using real data [ 14 ] . However, since LLMs are trained on texts written by humans, relying solely on real data may amplify societal bias and overlook fair outputs [ 15 ] . Therefore, a more holistic framework that considers both evaluation principles—whether a reference output is provided or not—can enable a deeper understanding of how LLMs behave in real-world applications. Stereotype bias and deviation bias can occur when LLM queries include explicit indications of gender, race, or religion. However, demographic attributes can also be inferred through implicit signals, such as names that are more likely to be associated with a certain gender or race. For example, Chen et al. [ 16 ] employ LLMs to rank the resumes of candidates for different residency programs. While GPT-4 favors Black and Hispanic candidates for multiple specialties when resumes explicitly mention the candidate’s race, it exhibits a much lower degree of racial bias when the candidate’s race is only indicated through their last name. Given that LLMs retain a memory of the user from the conversation history [ 17 ] , it is important to understand how LLMs may exhibit bias after learning more about the user through the usage of the model. The attributes of the user can either be provided during usage or inferred from other information, such as the name of the user. To this end, previous work has explored both implicit and explicit attributes in the contexts of gender, race, age, and disability [ 18 , 16 ] , showing homogeneity between prompts with implicit and explicit attributes. Inspired by these studies, we provide LLMs with prompts containing both implicit and explicit attributes, exploring the similarities and differences between implicit and explicit bias. Our study investigates bias by asking LLMs to generate profiles of individuals with information regarding their political affiliation, religion, sexual orientation, socioeconomic status, and occupation. Each individual’s gender, race, or age is indicated in the prompt explicitly (e.g., Hispanic male) or implicitly via name (e.g., Jose). The distribution of each demographic variable with respect to gender, race, or age is then calculated. Binomial tests are then performed between the proportions of texts generated for a specific group belonging to a certain demographic variable (e.g., the proportion of Hispanic texts that are Christian) and reference proportions based on real-world data. Understanding how LLMs are predisposed to inferring certain demographic attributes of users can help users to be wary of how LLM responses may be biased, developers to address these issues in the models, and organizations to understand the risks of using LLMs in commercial products. Results Politics Implicit (a) Aggregate results across all models for political affiliation with implicit gender input. (b) Aggregate results across all models for political affiliation with implicit ethnicity input. (c) Aggregate results across all models for political affiliation with implicit age input. Figure 1 : The political affiliation distributions for texts

Read the complete text