Bias is a Math Problem, AI Bias is a Technical Problem: 10-year Literature Review of AI/LLM Bias Research Reveals Narrow [Gender-Centric] Conceptions of “Bias”, and Academia-Industry Gap

Sourojit Ghosh, Kyra Wilson · 2025

Read Bias is a Math Problem, AI Bias is a Technical Problem: 10-year Literature Review of AI/LLM Bias Research Reveals Narrow [Gender-Centric] Conceptions of “Bias”, and Academia-Industry Gap by Sourojit Ghosh, Kyra Wilson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11067.txt.

Introduction

Sourojit Ghosh and Kyra Wilson review ten years of research on bias in artificial intelligence and large language models, in work published in 2025. They read 189 papers from four major venues in the field. Their aim is to map what bias research has covered, what it has left out, and how often its ideas reach real systems.

Most of the papers, they find, never define bias in human terms and give only a math or technical sense of the word. Nearly four in five focus on gender, while race, age, faith, and nationality get far less attention. Only about one paper in ten tells builders how to use the findings in practice. The authors call for wider coverage of groups outside the West.

Catalog record

CREATORSourojit Ghosh, Kyra Wilson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bias is a Math Problem, AI Bias is a Technical Problem: 10-year Literature Review of AI/LLM Bias Research Reveals Narrow [Gender-Centric] Conceptions of “Bias”, and Academia-Industry Gap Sourojit Ghosh, Kyra Wilson Abstract The rapid development of AI tools and implementation of LLMs within downstream tasks has been paralleled by a surge in research exploring how the outputs of such AI/LLM systems embed biases, a research topic which was already being extensively explored before the era of ChatGPT. Given the high volume of research around the biases within the outputs of AI systems and LLMs, it is imperative to conduct systematic literature reviews to document throughlines within such research. In this paper, we conduct such a review of research covering AI/LLM bias in four premier venues/organizations – *ACL, FAccT, NeurIPS, and AAAI – published over the past 10 years. Through a coverage of 189 papers, we uncover patterns of bias research and along what axes of human identity they commonly focus. The first emergent pattern within the corpus was that 82% (155/189) papers did not establish a working definition of “bias” for their purposes, opting instead to simply state that biases and stereotypes exist that can have harmful downstream effects while establishing only mathematical and technical definition of bias. 94 of these 155 papers have been published in the past 5 years, after Blodgett et al. 2020 ’s literature review with a similar finding about NLP research and recommendation to consider how such researchers should conceptualize bias, going beyond strictly technical definitions. Furthermore, we find that a large majority of papers – 79.9% or 151/189 papers – focus on gender bias (mostly, gender and occupation bias) within the outputs of AI systems and LLMs. By demonstrating a strong focus within the field on gender, race/ethnicity (30.2%; 57/189), age (20.6%; 39/189), religion (19.1%; 36/189) and nationality (13.2%; 25/189) bias, we document how researchers adopt a fairly narrow conception of AI bias by overlooking several non-Western communities in fairness research, as we advocate for a stronger coverage of such populations. Finally, we note that while our corpus contains several examples of innovative debiasing methods across the aforementioned aspects of human identity, only 10.6% (20/189) include recommendations for how to implement their findings or contributions in real-world AI systems or design processes. This indicates a concerning academia-industry gap, especially since many of the biases that our corpus contains several successful mitigation methods that still persist within the outputs of AI systems and LLMs commonly used today. We conclude with recommendations towards future AI/LLM fairness research, with stronger focus on diverse marginalized populations. 1 Introduction Though Artificial Intelligence (AI) is by no means a technology novel to the 21 st century, the development of the AI assistant ChatGPT powered by the Large Language Model (LLM) GPT-3 and its subsequent release for public usage in November 2022 spurred an era of promise and heralded the still-ongoing AI summer. As billions of dollars are continually poured across the world into the design and adoption of novel AI systems and tools leveraging multimodal LLMs in downstream tasks alongside the promulgation of new policies and legislations such as Executive Order 14179: Removing Barriers to American Leadership in Artificial Intelligence ( President of the United States 2025 ) in the US, there has also been a steep rise in LLM-related research throughout this period of time ( Pang et al. 2025 ) towards technical innovation and expanding the scope of tasks which AI systems and LLMs can accomplish. In addition, a significant proportion of this research has focused on the various ways in which the outputs of AI systems and LLMs exhibit bias towards traditionally marginalized populations ( Gupta et al. 2023 ; Wan et al. 2024 ) . This is a critical angle of research exploring the sociotechnicality of AI systems and LLM-powered tools, recognizing that the presence of negative stereotypes within their outputs against populations historically marginalized in society and oppressed through existing societal structures brings further harm upon them and exacerbates their marginalization. However, past research both in and adjacent to AI/LLMs, namely that of natural language processing (NLP), has demonstrated that researchers focusing on the nuances of biases within system outputs commonly display purely technical understandings of “bias” in their research ( Blodgett et al. 2020 , e.g,) . The combination of the rapid publication of AI/LLM bias research ( Pang et al. 2025 ) and a previously-recognized narrow conception of “bias” by technical researchers in related fields necessitates an exploration of the body of work in this field. In this paper, we perform a systematic literature review covering work published in four premier venues/organizations — *ACL, FAT*/FAccT, NIPS/NeurIPS, and AAAI — exploring emergent trends within bias and fairness research in AI systems and LLMs published within the last 10 years. Our coverage includes 189 full papers, curated through computational and manual screening, as we make the following three contributions to the field: 1. Firstly, we observe that 82% (155/189) papers studying “bias” in AI systems or LLMs do not actually define “bias”, and 74 of these 136 have been published in the last 5 years during the massive surge of LLM/AI fairness research. This is particularly concerning in light of Blodgett et al. 2020 ’s similar finding 5 years ago, as it is indicative that researchers in the field have not since taken their call to heart. The lack of a definition of “bias” in a paper studying “bias” poses real problems, and we provide recommendations for researchers to adopt more than mathematical conceptions of bias in their work. 2. We document a strong focus on ‘gender bias’ within the set of papers surveyed within our dataset, with 79.9% (151/189) papers demonstrating such a focus. Alongside a focus on gender (most prominently, the associations of gender with occupations, as in 19.6% or 37/189 papers), 30.2% (57/189) researchers also studied race/ethnicity bias (often equating the two), 13.2% (25/189) studied nationality bias, 20.6% (39/189) focused on age bias, and 19.1% (36/189) researchers worked on religion bias. Our findings highlight a concerning trend within the field that research around AI bias and fairness is often restricted to axes of identity perceived globally and interpreted through Western lenses (e.g., gender and occupation bias research often focuses based on US occupations data, examples of racial bias in word embeddings refers to African-Americans as criminals, etc.). This pattern has been alluded to in the past ( Dev et al. 2023 ; Ghosh 2024 ; Qadri et al. 2023 , e.g.,) , but not explicitly demonstrated through such a coverage of the field as ours. We highlight potential avenues to pursue research around historically marginalized communities often overlooked in AI-fairness research, to achieve stronger coverage of users and communities producing or consuming AI outputs. 3. Finally, it is evident across the corpus of papers that though researchers excel at innovative and effective debiasing techniques across the aforementioned axes of identity and providing technical solutions towards fairer AI systems, only 10.6% (20/189) papers contain any actionable implementation techniques of their findings within real-world AI systems. This is highlighted by an absence of potential design recommendations or mention of implementation procedures in papers, beyond making codebases and datasets publicly available for academic transparency. This

Read the complete text