Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory

Franziska Sofia Hafner, Ana Valdivia, Luc Rocher · 2025

Read Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory by Franziska Sofia Hafner, Ana Valdivia, Luc Rocher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14080.txt.

Introduction

Language models carry gender stereotypes, and a 2025 audit by Franziska Sofia Hafner, Ana Valdivia, and Luc Rocher argues the problem runs deeper than word links. Fixing which jobs get tied to which gender leaves untouched a deeper harm: models that treat gender as fixed biology can erase trans and diverse identities.

Guided by gender performativity theory, the team turned ideas about how language builds gender into testable checks. They then ran those checks on sixteen language models of varied designs, training data, and sizes. The audit shows how the models encode gender itself, not just pairs of words. The authors call on fairness research to widen its meaning of gender bias, since downstream harms range from misgendering users to wrong medical guesses.

Catalog record

CREATORFranziska Sofia Hafner, Ana Valdivia, Luc Rocher
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14080

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory Franziska Sofia Hafner, Ana Valdivia, Luc Rocher Abstract . Language models encode and subsequently perpetuate harmful gendered stereotypes. Research has succeeded in mitigating some of these harms, e.g. by dissociating non-gendered terms such as occupations from gendered terms such as ‘woman’ and ‘man’. This approach, however, remains superficial given that associations are only one form of prejudice through which gendered harms arise. Critical scholarship on gender, such as gender performativity theory, emphasizes how harms often arise from the construction of gender itself, such as conflating gender with biological sex. In language models, these issues could lead to the erasure of transgender and gender diverse identities and cause harms in downstream applications, from misgendering users to misdiagnosing patients based on wrong assumptions about their anatomy. For FAccT research on gendered harms to go beyond superficial linguistic associations, we advocate for a broader definition of ‘gender bias’ in language models. We operationalize insights on the construction of gender through language from gender studies literature and then empirically test how 16 language models of different architectures, training datasets, and model sizes encode gender. We find that language models tend to encode gender as a binary category tied to biological sex, and that gendered terms that do not neatly fall into one of these binary categories are erased and pathologized. Finally, we show that larger models, which achieve better results on performance benchmarks, learn stronger associations between gender and sex, further reinforcing a narrow understanding of gender. Our findings lead us to call for a re-evaluation of how gendered harms in language models are defined and addressed. Warning: This article discusses the erasure and pathologization of transgender and gender diverse identities in academic literature and in language models. Conference: The 2025 ACM Conference on Fairness, Accountability, and Transparency; June 23–26, 2025; Athens, Greece The 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25), June 23–26, 2025, Athens, Greece DOI: 10.1145/3715275.3732112 ISBN: 979-8-4007-1482-5/2025/06 CCS: Computing methodologies Natural language processing CCS: Applied computing Sociology † † cc-license: by 1. Introduction Language plays a key role in constructing, reinforcing, and subverting gender norms ( Baker, 2008 ) . The English language, for example, constructs gender by assigning the binary categories ‘boy’ and ‘girl’ to babies at birth, reinforces gender through phrases such as ‘boys will be boys’ and ‘throw like a girl’, and subverts gender through terms not fitting into the ‘boy/girl’ binary, such as the use of ‘they/them’ as singular pronouns. The more frequently some gender norm is expressed through language, the more it is normalized ( Butler, 2006 ) . Conversely, gendered experiences not often expressed through language might be perceived as abnormal, unacceptable, or even unimaginable. Language models (LMs) encode these norms from the books, online news articles, and online forums that make up their training data ( Birhane et al., 2024 ; Dodge et al., 2021 ; Steed et al., 2022 ; Feng et al., 2023 ; Köksal et al., 2023 ) . Encoding human biases and their representation in popular culture, LM-based technologies can then persistently reproduce prejudice against sexual orientation or gender identity. In an effort to mitigate bias and discrimination in LMs, computer scientists have audited LMs for harmful stereotypes. This research has contributed to understanding and reducing these harms ( Gallegos et al., 2024 ) . For example, researchers have used large dictionaries of ‘bias probes’ and ‘prefix templates’ to elicit biased sentence completions ( Solaiman et al., 2019 ; Sheng et al., 2019 ) . Further research on downstream applications, notably in HR and recruitment as well as in clinical and medical applications, showed that LMs consistently stereotype certain races, ethnicities, and genders. Subsequently, new approaches have been developed which target training datasets (e.g. by re-balancing data with pre-defined list of biased word pairs such as ‘he/she’ and ‘king/queen’ ( Liu et al., 2021 ) ), learning phase (e.g. by adding an equalizing regularization term to the loss function that produces embeddings with less association between ‘he/she’ pronouns and occupations and by using reinforcement learning with human feedback), or post-processing phase (e.g. by setting prompts that help avoid biased sentence completion, or replacing stereotypical keywords) ( Gallegos et al., 2024 ) . However, while debiasing efforts reduce easy-to-see forms of bias in simple benchmarks, they fail to address more complex and deeply ingrained associations ( Wang et al., 2023 ; Ovalle et al., 2024 ) , especially in adversarial settings ( Qi et al., 2023 ) or in Chain of Thought settings that prompt models to ‘think’ ( Shaikh et al., 2023 ) , both circumventing debiasing efforts. Hofmann et al. (2024) found that such efforts can successfully reduce negative overt racist stereotypes but exacerbate covert stereotypes, in the form of dialect prejudice. For example, some models are optimized to reduce the association of ‘he/she’ pronouns or ‘black/white’ racial categories with occupations. This may not affect cases where gender is inputed from pronouns other than ‘he/she’, or race from racialised terms other than ‘black/white’, e.g. through implicit information such as dialect or writing style. Moreover, models optimized according to specific categories such as ‘man’ and ‘woman’ risk essentializing complex social identities such as gender into a narrow set of biological attributes or culturally dominant norms, by reinforcing the idea that these categories are fixed, universal, or exhaustive. Addressing bias and discrimination in LMs requires moving beyond aligning models with pre-existing human preferences, by leveraging decades of critical research from the social sciences on how social characteristics are and should be represented in language ( Blodgett et al., 2020 ) . Research in fields such as gender studies, sociolinguistics, feminist studies, and queer studies highlights the importance of language not merely in representing gender, but also in constructing, reinforcing, and subverting gender norms ( Baker, 2008 ) . Critically, they show that gendered harms often arise from the construction of gender itself, instead of from stereotypical linguistic associations. For example, gender and sex are frequently linked narrowly in language ( man linked to male , woman linked to female ). This strong association can be harmful both to cisgender people (e.g., if the ability to conceive is narrowly tied to womanhood, this is harmful to cisgender women wo do not want to or cannot get pregnant) and to transgender and gender diverse people (e.g., a transgender woman who cannot get pregnant might be excluded from womanhood on this basis). Additionally, by defining classifications of gender (e.g., man and woman ), language can contribute to the erasure of gender identities that are not considered in such classifications, such as transgender and gender diverse people. We build on these insights from gender studies, and apply them to the study of LMs. We derive three main insights on the importance of language from Butler’s works on gender performativity and apply them to LMs: (a) distinction between gender and sex (LMs should not essentialize gender by conflating it with biological sex), (b) meaningful embeddings of diverse genders (LMs should provide meaningful embeddings for all gender identity terms), and (c) non-stereotypical representations (LMs should not encode harmful stereotypes about any gender identity terms). We test how 16 LMs, including versions

Read the complete text