From Structured Prompts to Open Narratives: Measuring Gender Bias in LLMs Through Open-Ended Storytelling
Evan Chen, Run-Jun Zhan, Yan-Bai Lin, Hung-Hsuan Chen · 2025
Read From Structured Prompts to Open Narratives: Measuring Gender Bias in LLMs Through Open-Ended Storytelling by Evan Chen, Run-Jun Zhan, Yan-Bai Lin, Hung-Hsuan Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15904.txt.
Introduction
Do chat models lean on gender clichés when they tell tales? Evan Chen, Run-Jun Zhan, Yan-Bai Lin, and Hung-Hsuan Chen tested ten big language models in 2025 by asking for free-form tales and counting who shows up in which jobs. The head of the text gives the piece a sharper title, More Women, Same Stereotypes, while the catalog lists a plainer one. The count gave a surprise: women fill more roles than real work data would suggest, likely a side effect of the fine-tuning that polishes these models.
Yet the jobs women got still matched old clichés more than real labor counts. More women on the page did not mean fairer casting. The team warn that fixes which boost one count can plant fresh bias while they patch the old. They post their prompts and the tales the models wrote for all to check.
Catalog record
| CREATOR | Evan Chen, Run-Jun Zhan, Yan-Bai Lin, Hung-Hsuan Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15904
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Structured Prompts to Open Narratives: Measuring Gender Bias in LLMs Through Open-Ended Storytelling Evan Chen, Run-Jun Zhan, Yan-Bai Lin, Hung-Hsuan Chen Abstract Large Language Models (LLMs) have revolutionized natural language processing, yet concerns persist regarding their tendency to reflect or amplify social biases present in their training data. This study introduces a novel evaluation framework to uncover gender biases in LLMs, focusing on their occupational narratives. Unlike previous methods relying on structured scenarios or carefully crafted prompts, our approach leverages free-form storytelling to reveal biases embedded in the models. Systematic analyses show an overrepresentation of female characters across occupations in six widely used LLMs. Additionally, our findings reveal that LLM-generated occupational gender rankings align more closely with human stereotypes than actual labor statistics. These insights underscore the need for balanced mitigation strategies to ensure fairness while avoiding the reinforcement of new stereotypes. 1 Introduction Large Language Models (LLMs) have rapidly become integral to natural language processing and decision support. Despite their capabilities, these models may inadvertently preserve or even amplify social biases rooted in training data. Accurately quantifying these biases is essential to ensure their fair and responsible deployment. Previous studies have used various methods to assess LLMs’ gender biases in professional contexts. Some approaches involve structured decision-making tasks, such as choosing candidates from lists of names tied to different demographic groups ( Nghiem et al. 2024 ; An et al. 2024 ) . Other strategies test linguistic biases using prompts with ambiguous pronouns, e.g., “The doctor phoned the nurse because she was late” ( Kotek et al. 2023 ) . While effective, these methods are limited to predefined scenarios, risk oversimplifying biases, and may reveal the evaluation’s intent, allowing LLMs to adapt strategically. To address these shortcomings, we propose a simple evaluation strategy that avoids predefined scenarios and conceals the intent of the test. Specifically, we request LLMs to generate a story opening and setting for a character in a specific occupation and analyze the gender cues in the output, such as names and pronouns (i.e., she or he). We execute the exact prompt multiple times for each occupation and evaluate the gender distribution across repeated outputs. This method eliminates the need for strictly designed scenarios or carefully tailored prompts, allowing biases to appear naturally within the context of free-form storytelling. It avoids explicitly directing the model’s attention toward gender-related considerations. Thus, it provides an authentic and comprehensive view of occupational gender stereotypes. Based on this new evaluation strategy, we have the following observations. First, LLMs exhibit an apparent overrepresentation of female characters in their generated stories, with 58 of 100 occupations showing female dominance ( 80\% or more characters are female) across the six LLMs tested. Interestingly, this female dominance is not limited to traditionally stereotypical female-associated occupations. A possible reason for this pattern is that LLM service providers may have intentionally adjusted the models to increase female representation, perhaps aiming to counteract historical gender biases in language data. This intentional weighting could lead to overrepresenting female characters in various occupations, not just those traditionally associated with women. Second, the rankings of occupations by gender ratios in LLM outputs align more closely with the perceived gender associations of occupations (as measured by human ratings) than the actual gender distributions in the labor force. This indicates that LLMs are more likely to mirror societal perceptions of gender roles than real-world demographic distributions, potentially amplifying biases rooted in societal stereotypes. Although this observation may seem contradictory to the first conclusion, it is not. The increased representation of females across all categories appears to lead many occupations to become female-dominated. However, when occupations are ranked by gender composition ratios, the resulting sequence still strongly aligns with human perceptions of gender associations. The rest of the paper is organized as follows. In Section 2 , we review previous work on quantifying the bias of LLMs. Section 3 introduces our methodology for evaluating LLM gender biases in occupations. Section 4 presents the discoveries based on our evaluation. Finally, we discuss and conclude the work in Section 5 . 2 Related Work Several studies have investigated the presence and impact of demographic biases in LLMs. These studies employ a range of methodologies and focus on different aspects of biases. 2.1 Evaluating Bias by LLMs’ Decisions on Strictly Designed Scenarios Several studies have asked LLMs to make decisions or recommendations, allowing researchers to assess biases in these results ( Nghiem et al. 2024 ; An et al. 2024 ; Salinas et al. 2023 ; Haim et al. 2024 ) . For example, Nghiem et al. 2024 prompted LLMs to select the “best” candidates from lists of names associated with different racial and gender groups in 40 occupations. Their findings revealed a significant preference for white male names. An et al. 2024 similarly investigated the impact of applicant names on LLM-generated hiring decisions. They manipulated applicant names in the prompts to measure the influence of perceived ethnicity and gender on acceptance rates. Their results showed evidence of discrimination against Hispanic applicants, particularly males. Although these studies effectively demonstrate the presence of name-based biases, they are limited to strictly designed scenarios and may not be generalized to other occupational contexts or broader real-world applications. 2.2 Evaluating Bias by LLMs’ Word Choices on Carefully Designed Prompts Another line of research has focused on analyzing the linguistic features of LLM-generated text to uncover biases ( Wan et al. 2023 ; Kotek et al. 2023 ; Lee et al. 2024 ; Soundararajan and Delany 2024 ; Kumar et al. 2024 ; Bas 2024 ) . For instance, Wan et al. 2023 analyzed gender biases in LLM-generated recommendation letters by examining both the language style and lexical content used to describe male and female candidates. They revealed significant gender bias in ChatGPT and Alpaca, with female candidates often described using more communal descriptions (e.g., kind, great to work with) while male candidates were described using more agentic descriptions (e.g., expert, a standout in the industry). Kotek et al. 2023 used ambiguous sentences with gendered occupations and pronouns to assess LLMs’ reliance on gender stereotypes and analyze their reasoning. They found that LLMs frequently rely on gender stereotypes to resolve pronouns, revealing an implicit bias in the models’ understanding of gender and occupations. And finding that LLMs often rationalize these biases with flawed logic. Lee et al. 2024 investigated the inherent biases and toxicity of LLMs in the Korean social context. They assign various personas to LLMs and evaluate the toxicity of the word choices. They found that specific persona-issue combinations could yield harmful content, indicating that LLMs can reflect and amplify social biases existing in particular cultural contexts. Unlike previous approaches, we prompt LLMs to generate stories about a variety of professions and analyze the gender ratio of the characters in the stories. This method allows LLMs to describe open-ended stories rather than in restricted scenarios (e.g., Nghiem et al. 2024 and An et al. 2024 ) and eliminates the need for meticulously designed prompt sentences (e.g., Kotek et al. 2023 ). Thus, it has the advantage of capturing more natural and authentic biases embedded in the model’s narrative style, providing a broader and less constrained perspective on occupational gender