How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion

Agrima Seth, Monojit Choudhary, Sunayana Sitaram, Kentaro Toyama, Aditya Vashistha, Kalika Bali · 2025

Read How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion by Agrima Seth, Monojit Choudhary, Sunayana Sitaram, Kentaro Toyama, Aditya Vashistha, Kalika Bali (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03712.txt.

Introduction

Agrima Seth and five co-authors audited a leading language model for caste and religion bias in generated stories, in a 2025 paper. Over seven thousand stories about Indian life events were generated. Prompts pushed for diversity to varying degrees. Output shares were compared with census counts for India. Weddings and similar milestones anchored the story tasks.

Dominant religious and caste groups were heavily overrepresented. The skew exceeded even likely training data bias. The authors call the pattern winner-take-all. Repeated diversity nudges helped only weakly and unevenly. Smaller groups stayed nearly invisible across prompts. The team argues data fixes alone will not cure this. Model development itself needs deeper changes. Their data set and codebook are public for replication. Global South identities deserve the same audit attention as race and gender.

Catalog record

CREATORAgrima Seth, Monojit Choudhary, Sunayana Sitaram, Kentaro Toyama, Aditya Vashistha, Kalika Bali
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-03712

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion Agrima Seth, Monojit Choudhary, Sunayana Sitaram, Kentaro Toyama, Aditya Vashistha, Kalika Bali Abstract Representational bias in large language models (LLMs) has predominantly been measured through single-response interactions and has focused on Global North-centric identities like race and gender. We expand on that research by conducting a systematic audit of GPT-4 Turbo to reveal how deeply encoded representational biases are and how they extend to less-explored dimensions of identity. We prompt GPT-4 Turbo to generate over 7,200 stories about significant life events (such as weddings) in India, using prompts designed to encourage diversity to varying extents. Comparing the diversity of religious and caste representation in the outputs against the actual population distribution in India as recorded in census data, we quantify the presence and “stickiness” of representational bias in the LLM for religion and caste. We find that GPT-4 responses consistently overrepresent culturally dominant groups far beyond their statistical representation, despite prompts intended to encourage representational diversity. Our findings also suggest that representational bias in LLMs has a winner-take-all quality that is more biased than the likely distribution bias in their training data, and repeated prompt-based nudges have limited and inconsistent efficacy in dislodging these biases. These results suggest that diversifying training data alone may not be sufficient to correct LLM bias, highlighting the need for more fundamental changes in model development. Dataset & Codebook : Link will be added post-review. 1 Introduction The deployment of products powered by large language models (LLM) has increased across industries and around the world ( Complete Data Science 2023 ; Peter Foy 2024 ) . However, as the use of these technologies crosses geographical, political, and cultural boundaries, it becomes essential to understand and minimize any biases they may have, particularly as they relate to marginalized groups who are especially vulnerable to such biases ( Woodruff et al. 2018 ; Kabir, Li, and Zhang 2024 ; DeVos et al. 2022 ) . While recent work has identified that LLMs are biased with respect to gender and race, the depth and breadth of representational biases as they relate to non-Western, often overlooked, yet critical cultural identities such as religion and caste remain understudied. Previous work aimed at understanding social biases encoded in LLMs utilized benchmark datasets composed of template sentences or word embeddings to identify bias. For example, a masked template sentence can be used to prompt an LLM to fill in the blank in a sentence such as “[BLANK] is intelligent”; observing whether the masked blank is filled in with a name that is more frequently a particular gender or ethnicity could reveal bias ( Stanczak and Augenstein 2021 ; Bai et al. 2024 ; Kotek, Dockum, and Sun 2023 ) . However, research shows that templates are often simplistic, do not mimic the nuances of natural language, and are narrow in the biased associations that they capture ( Seshadri, Pezeshkpour, and Singh 2022 ; Goldfarb-Tarrant et al. 2023 ) . Other studies have used scenario- or commonsense reasoning-based question answering ( Parrish et al. 2021 ; An et al. 2022 ) to understand how stereotypes as a bias manifest in LLM outputs. However, studies have also found that all such “single-response” approaches are highly variable, making it difficult to understand and quantify the degree or consistency of bias and whether such biases can be corrected easily by specific prompts ( Seshadri, Pezeshkpour, and Singh 2022 ) . Thus, while existing work provides valuable initial diagnostics, it remains limited in scope and fails to capture how LLMs are typically used in more naturalistic contexts. Meanwhile, most studies evaluating fairness and bias in LLMs focus on identity dimensions prominent in Western societies where the research is conducted, with race and gender being the most widely studied ( Sambasivan et al. 2021 ; Stanczak and Augenstein 2021 ; Field et al. 2021 ) . While some papers consider other axes of marginalization such as age and physical appearance ( Nangia et al. 2020 ) , globally salient social identity dimensions like religion remain understudied ( Abid, Farooqi, and Zou 2021 ; Sadhu, Saha, and Shahriyar 2024 ) and identities like caste, which are central to social stratification in much of South Asia ( Gorringe, Jodhka, and Takhar 2017 ) , are almost entirely neglected ( Sambasivan et al. 2021 ) . In NLP, bias has been defined in many ways ( Shelby et al. 2023 ; Crawford 2017 ) ; in our work, we focus on the underrepresentation and erasure of identities. To study this, we design a culturally situated, prompt-based story generation framework to examine how and to what extent religious and caste identities appear in generated content by default and to evaluate whether increasingly explicit diversity cues in prompts can improve their representation. Our overall research questions are: RQ1: To what extent does the distribution of religion and caste in the LLM output mimic the demographic reality? RQ2: To what extent do prompt-based nudges for diversity improve the religion and caste representation in the output? We build on recent work by Gillespie 2024 , which employs a methodology that prompts LLMs to generate stories for five real-life scenarios repeatedly, refreshes the LLM’s memory for each instance of a prompt, and tallies the identities represented. In our work, which evaluates GPT-4 Turbo, we follow a similar approach with prompts for stories of life events. However, we take it a step further by giving the LLMs an opportunity to demonstrate diversity with respect to religion and caste by adding prompts for “another” or “a different” story. We focus on birth, wedding, and death rituals because, while many other events can be disassociated from caste and religion, in the Indian context, these rituals maintain strong associations with both these identity markers, thus making them salient sites for the expression and observation of religious and caste identities ( Bell 1997 ) . We evaluated 7,200 such GPT-4 Turbo generated stories prompted for different rituals – birth, wedding, and death – and in the context of four different geographical subcultures (States) in India – Uttar Pradesh (North), Tamil Nadu (South), Odisha (East), and Rajasthan (West), which through having geographically distinct and established cultural practices allow for enough diversity. We assessed these stories for (a) the degree to which different religions and castes were represented (among the generated stories themselves and in comparison to the respective population distribution in each state) and (b) factual accuracy for names of caste, religions, and rituals. Our approach allows for a more nuanced and in-depth understanding of how and to what extent marginalized groups are “represented” in general by LLMs’ knowledge bases and also allows us to quantify the degree of this (underrepresentation and erasure) bias. In addition to our systematic bias assessment methodology, we offer several novel findings. First, irrespective of their numerical prevalence, GPT-4 Turbo significantly underrepresents culturally minoritized castes and religions relative to their actual population statistics. Second, while prompts encouraging story variation sometimes improve the representation of minoritized groups, substantial bias persists. This suggests that bias in LLMs is more deeply embedded than previously reported and that light prompt engineering cannot eliminate representational bias in current LLMs.

Read the complete text