Prompt Fairness: Sub-group Disparities in LLMs

Meiyu Zhong Noel Teku Ravi Tandon · 2025

Read Prompt Fairness: Sub-group Disparities in LLMs by Meiyu Zhong Noel Teku Ravi Tandon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19956.txt.

Introduction

Meiyu Zhong, Noel Teku, and Ravi Tandon study prompt fairness, a gap in how language models treat different users. The same question, phrased in different styles by different groups, can draw answers of uneven quality. The authors set out to measure this disparity and then reduce it.

They propose information-based measures covering two sides: how much answers vary inside one subgroup, and how far groups diverge from each other. Tests show some subgroups face both higher internal variation and greater distance from others, a sign of structural inequity. Two fixes, majority voting over several replies and prompt neutralization, steady the outputs and shrink the gaps between groups in their experiments.

Catalog record

CREATORMeiyu Zhong Noel Teku Ravi Tandon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19956

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prompt Fairness: Sub-group Disparities in LLMs Thanks: This work was supported by US NSF under Grants CCF-2100013, CNS-2209951, CNS-2317192; by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing under Award DE-SC-ERKJ422; and by NIH under Award R01-CA261457-01A1. Meiyu Zhong Noel Teku Ravi Tandon Abstract Large Language Models (LLMs), though shown to be effective in many applications, can vary significantly in their response quality. In this paper, we investigate this problem of prompt fairness : specifically, the phrasing of a prompt by different users/styles, despite the same question being asked in principle, may elicit different responses from an LLM. To quantify this disparity, we propose to use information-theoretic metrics that can capture two dimensions of bias: subgroup sensitivity —the variability of responses within a subgroup—and cross-group consistency —the variability of responses across subgroups. Our analysis reveals that certain subgroups exhibit both higher internal variability and greater divergence from others. Our empirical analysis reveals that certain demographic subgroups experience both higher internal variability and greater divergence from others, indicating structural inequities in model behavior. To mitigate these disparities, we propose practical interventions, including majority voting across multiple generations and prompt neutralization , which together improve response stability and enhance fairness across user populations. In the experiments, we observe clear prompt-sensitivity disparities across demographic subgroups: before mitigation, cross-group divergence values reach 0.28 and typically fall in the 0.14–0.22 range. After applying our neutralization and multi-generation strategy, these divergences consistently decrease, with the largest gap reduced to 0.22 and many distances falling to 0.17 or below, indicating more stable and consistent outputs across subgroups. I Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, including question answering, summarization, and reasoning. However, recent studies have shown that semantically equivalent prompts can elicit divergent answers from state-of-the-art LLMs [ 12 ] [ 34 ] , raising concerns about robustness and fairness in real-world applications. This phenomenon, known as prompt sensitivity, poses a critical challenge to the reliability and consistency of LLM-generated information. While these issues have been studied for general variations of the prompt, a more pressing issue is if the LLM can still answer a query correctly across members of different subgroups (e.g. race, gender) who may phrase the query differently. Motivation: Figure 1 shows a representative example in how a particular phrasing of a prompt (according to different demographics) can affect an LLM’s sentiment classification ability. We take an entry from the IMDB dataset, which contains movie reviews and a label showing the sentiment of the review (i.e. positive or negative). We re-stylize the same review in three different tones (white male, black male, and valley girl) expressed by three different subgroups using different dialects and then pass to the same LLM Llama-8B. The figure shows that the sentiment predictions by the LLM were inconsistent across subgroups, which indicates that the phrasing of a prompt by a particular subgroup can elicit a different response from an LLM compared to other subgroups. Studying how demographic attributes and prompting styles affect LLM response quality is an emerging field in the literature [ 8 , 2 ] . These disparities in response quality raise important concerns about fairness, accessibility, and the equitable deployment of LLMs in real-world settings. As LLMs are integrated into high-stakes domains like education, healthcare, and legal assistance, understanding and mitigating the ways in which demographic variation impacts response behavior is critical for building inclusive and trustworthy systems. Despite the growing literature on this topic, there is a lack of a fine-grained approach to quantify and assess the impact of LLM sensitivity and consistency across sub-groups. Fig. 1: Showing three variations of a review from three subgroups passed to an LLM for sentiment classification. The LLM responds differently based on the tone of the prompt variation, which reveals the bias through prompt rephrasing by different groups (white male, black male, valley girl). Overview of Contributions: The primary objective of this work is to examine whether LLMs exhibit consistent sensitivity to prompt variations across different user subgroups. Specifically, we ask: “Do LLMs respond differently when the same query is phrased in the linguistic styles of different demographic groups? How can we quantify this bias, and how might we mitigate it?” To address these questions, we investigate the phenomenon of prompt fairness by introducing metrics for subgroup sensitivity and cross-group pairwise consistency, focusing on demographic subgroups (e.g., white males, Black females). Our findings indicate that subtle variations in tone and phrasing—often reflective of subgroup-specific linguistic styles—can significantly affect LLM response quality. To empirically study this behavior, we develop a controlled evaluation pipeline that generates subgroup-conditioned paraphrases, removes demographic cues through prompt neutralization, and embeds model outputs into a semantic space to quantify divergence across groups. Using information-theoretic metrics, we find that LLM outputs vary non-trivially across demographic subgroups, with cross-group divergences reaching values up to 0.28 in the absence of intervention. We further demonstrate that targeted mitigation strategies, including majority voting across multiple generations and structured prompt neutralization, substantially reduce these disparities, lowering peak divergence values to approximately 0.22 and yielding more stable responses across groups. Taken together, this work contributes a rigorous framework for evaluating prompt-induced disparities in LLM behavior, provides quantitative evidence of subgroup-specific sensitivity to linguistic variation, and proposes effective mechanisms for reducing such disparities. These findings underscore the importance of robustness to prompt variation in equitable model deployment and offer methodological guidance for future research at the intersection of language variation, fairness, and foundation model reliability. II Related Works Our work builds on a rich body of research in algorithmic fairness and equitable machine learning. A substantial literature has examined fairness in supervised learning systems through statistical criteria such as demographic parity and equalized odds [ 11 , 23 , 29 , 14 , 28 ] , representation learning for fairness [ 25 , 18 ] , and robustness to distribution shift [ 10 , 31 , 5 , 26 , 30 ] . More recently, fairness concerns have extended to large generative and language models, including bias measurement [ 21 ] and mitigation techniques spanning data interventions, model regularization, and prompt-based strategies [ 15 , 32 , 3 ] . There have been a growing number of works in the literature studying the prompt sensitivity of LLMs. [ 13 , 20 ] showed that the order and formatting of a prompt can have significant impacts on the accuracy of a LLM respectively. Mechanisms for measuring prompt sensitivity have been proposed in [ 4 , 34 ] . [ 12 ] measured the sensitivity and consistency of LLMs performing classification when given variations of prompts. Additionally, there has been growing work prompting LLMs with certain personas and studying their effects on generated responses. [ 6 ] showed that LLMs embodying personas of minority classes (marked groups) used potentially harmful/descriptive language to describe their identity while personas of the majority class (unmarked groups) expressed themselves with neutral, harmless words. [ 17 ] studied how attributes of persona prompting strategies (role adoption and demographic priming) can affect the ability of the LLM to emulate a given persona in its responses. [ 22 ] showed that when prompts from evaluation datasets are rephrased by

Read the complete text