Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation
Petter Törnberg · 2026
Read Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation by Petter Törnberg (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13891.txt.
Introduction
Törnberg tests whether large language models bake racial stereotypes into text annotation. The study covers 19 models and over 4 million judgments in two experiments. Names linked to Black people draw more aggressive and gossipy ratings from most models. Asian names draw higher intelligence ratings but lower confidence ratings. The same sentence in African American Vernacular English is judged less professional and more toxic than in standard English. One exception appears in hiring scores, where fine-tuning seems to overcorrect.
Catalog record
| CREATOR | Petter Törnberg |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47997 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13891
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation Petter Törnberg Abstract Large language models (LLMs) are increasingly used for automated text annotation in tasks ranging from academic research to content moderation and hiring. Across 19 LLMs and two experiments totaling more than 4 million annotation judgments, we show that subtle identity cues embedded in text systematically bias annotation outcomes in ways that mirror racial stereotypes. In a names-based experiment spanning 39 annotation tasks, texts containing names associated with Black individuals are rated as more aggressive by 18 of 19 models and more gossipy by 18 of 19. Asian names produce a bamboo-ceiling profile: 17 of 19 models rate individuals as more intelligent, while 18 of 19 rate them as less confident and less sociable. Arab names elicit cognitive elevation alongside interpersonal devaluation, and all four minority groups are consistently rated as less self-disciplined. In a matched dialect experiment, the same sentence is judged significantly less professional (all 19 models, mean gap − 0.774 -0.774 ), less indicative of an educated speaker ( − 0.688 -0.688 ), more toxic (18/19), and more angry (19/19) when written in African American Vernacular English rather than Standard American English. A notable exception occurs for name-based hireability, where fine-tuning appears to overcorrect, systematically favoring minority-named applicants. These findings suggest that using LLMs as automated annotators can embed socially patterned biases directly into the datasets and measurements that increasingly underpin research, governance, and decision-making. Keywords: large language models; text annotation; racial bias; stereotype; African American Vernacular English; algorithmic discrimination 1 Introduction Large language models (LLMs) are rapidly becoming standard tools for automated text annotation across the social and behavioral sciences [ 14 , 26 , 20 , 2 ] . Instruction-tuned models can perform annotation tasks in a zero-shot setting—guided only by a natural-language prompt rather than task-specific training data—and are increasingly used to label texts for sentiment, political opinion, personality traits, toxicity, and many other attributes [ 6 , 9 ] . Similar systems are also deployed commercially for tasks such as automated resume screening and hiring decisions that affect millions of people. This rapid adoption has been accompanied by growing concerns that LLMs absorb and reproduce the racial and ethnic stereotypes present in their training data [ 13 ] . Prior work shows that language models encode anti-Muslim bias in text completion [ 1 ] , that NLP classifiers disproportionately flag African American Vernacular English (AAVE) as toxic [ 24 , 5 , 10 ] , and that contemporary LLMs produce racially disparate evaluations when dialect signals racial identity [ 16 ] . Similar concerns about bias in automated decision systems have been widely documented across machine learning applications [ 8 ] . Yet most existing studies examine bias in generative outputs or in single-task classification settings. Whether—and how—social biases shape zero-shot annotation outcomes remains largely unexplored. We address this gap with a large-scale audit of 19 LLMs spanning nine organizations across the United States, Europe, and China. The study combines two complementary experimental paradigms. In Experiment 1, racially associated names were embedded in 211 biographical text vignettes, enabling measurement of annotation gaps across 39 binary classification tasks. In Experiment 2, we presented matched texts in Standard American English (SAE) and African American Vernacular English (AAVE), identical in propositional content and differing only in dialect, and measured annotation gaps across 16 tasks. Together, this design enables exploring how different types of implicit textual author identity cues shape LLM annotation across a wide range of dimensions. Across 19 LLMs, we show that matched identity cues embedded in otherwise similar texts—racialized first names and dialect—systematically shift zero-shot annotation outputs in stereotype-consistent directions. Names associated with different ethnic groups trigger coherent attribution profiles across character judgments, competence evaluations, and employability assessments. When identity is conveyed through dialect rather than name, the pattern shifts to near-universal disadvantage: across all tested models, the same text written in AAVE is consistently judged as less professional, less educated, more angry, and more toxic than an equivalent SAE version. A striking exception appears in name-based hireability judgments, where models systematically favor minority-named applicants. This reversal is consistent with targeted overcorrection by instruction tuning, while broader stereotype structures remain intact. Figure 1: Annotation gap architecture across all tasks and ethnic groups. Each panel shows one minority group. Points represent individual model gap estimates (minority minus White yes-rate); the thick vertical line indicates the cross-model mean and shading shows \pm 1 SD. Tasks are ordered by mean gap within each panel. Positive values indicate that the minority group is rated higher than White-named individuals; negative values indicate a penalty. Self-discipline (consistently at the lower end of all four panels) is the most uniformly penalized attribute across groups. Hireability (consistently at the upper end) is the most uniformly elevated. Arab names show the greatest within-task variation, reflecting simultaneous elevation on cognitive attributes and devaluation on interpersonal dimensions. Results Names shape annotation across many dimensions Across 39 annotation tasks, 19 models, and four minority ethnic groups, the names paradigm produced 116 FDR-corrected significant effects (Arab: 37; Asian: 31; Black: 30; Hispanic: 18), with directional tendencies extending across many non-significant cells. The patterns are not uniform: each group exhibits a recognizable stereotype profile that is broadly consistent across models (Fig. S1). Stereotype-consistent character attributions. Black and Arab names trigger consistent negative character attributions. Texts featuring Black names are rated as more aggressive by 18 of 19 models (cross-model mean gap +0.012 ; 1 FDR-significant) and more gossipy by 18 of 19 ( +0.019 ; 2 FDR-significant). Arab names are rated as lazier by 15 of 19 models ( +0.005 ) and as more emotional by 16 of 19 ( +0.019 ). These negative attributions are not offset by parallel elevations on positive character dimensions: Arab-named individuals are rated as less warm by 16 of 19 models ( -0.024 ) and as less good communicators by 17 of 19 ( -0.024 ), while Black-named individuals are rated as less warm by 18 of 19 ( -0.023 ). Together, these patterns indicate that instruction tuning has not removed stereotype-consistent character attribution from annotation outputs (Fig. 1 ). The Asian bamboo-ceiling profile. Asian names produce a sharply differentiated attribution pattern. On cognitive and creative dimensions they are consistently elevated: 17 of 19 models rate Asian-named individuals as more intelligent (mean +0.043 ; 4 FDR-significant) and 15 of 19 as more creative ( +0.026 ; 2 FDR-significant). On agentic and social dimensions the pattern reverses. Eighteen of 19 models rate Asian-named individuals as less confident (mean -0.036 ; 3 FDR-significant), less sociable ( -0.040 ; 1 FDR-significant), and less warm ( -0.025 ). Leadership is penalized by 17 of 19 models ( -0.032 ; 2 FDR-significant). This cognitive-elevation–agentic-penalty dissociation mirrors the “bamboo ceiling” documented in human studies of Asian stereotypes in professional contexts [ 3 ] , indicating that LLMs reproduce this multidimensional stereotype structure in annotation tasks. Hispanic poverty attribution and leadership penalty. Hispanic names produce a similar mixed pattern. Models systematically elevate cognitive assessments—17 of 19 rate Hispanic-named individuals as more intelligent (mean +0.036 ; 1 FDR-significant)—while simultaneously attributing economic disadvantage: 18 of 19 models rate Hispanic-named individuals as more likely to be poor ( +0.022